[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:34 dist-test-slave-8qt6 krb5kdc[21283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:34 dist-test-slave-8qt6 krb5kdc[21283](info): set up 2 sockets
Sep 25 14:53:34 dist-test-slave-8qt6 krb5kdc[21283](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:53:35.060796 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:35.172676 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38419
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:38419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:35.250670 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:35.250874 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:35.250895 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:35.252074 21296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:35.252115 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:35.252130 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:35.252175 21296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:35.252203 21296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:35.253643 21296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:38419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:35.253892 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:35.254114 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:35.259335 21296 server_base.cc:1034] running on GCE node
W20260925 14:53:35.259308 21302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:35.259308 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:35.259292 21301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:35.259732 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:35.259999 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:35.261142 21296 hybrid_clock.cc:648] HybridClock initialized: now 1790348015261123 us; error 34 us; skew 500 ppm
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:35.263495 21296 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:35.263931 21296 webserver.cc:460] Webserver started at http://127.20.199.62:35537/ using document root <none> and password file <none>
I20260925 14:53:35.264161 21296 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:35.264228 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:35.264375 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:35.265359 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "fc17b409427d4eb2af7754b3f2502901"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "36dce47eae39fde483fdae1645744385"
server_key_iv: "babee87591750a4ab84e529f0e8f7765"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.265779 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "fc17b409427d4eb2af7754b3f2502901"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "36dce47eae39fde483fdae1645744385"
server_key_iv: "babee87591750a4ab84e529f0e8f7765"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.267050 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:35.267738 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.267900 21296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:35.268002 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "fc17b409427d4eb2af7754b3f2502901"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "36dce47eae39fde483fdae1645744385"
server_key_iv: "babee87591750a4ab84e529f0e8f7765"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.268071 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:35.285668 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:35.287403 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:35.287556 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:35.291352 21296 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:38419
I20260925 14:53:35.291383 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:38419 every 8 connection(s)
I20260925 14:53:35.291746 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:35.292583 21364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:35.294755 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901: Bootstrap starting.
I20260925 14:53:35.295305 21364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:35.295589 21364 log.cc:824] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:35.296224 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901: No bootstrap required, opened a new log
I20260925 14:53:35.297017 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 21296
I20260925 14:53:35.297127 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:35.297896 21276 external_mini_cluster.cc:1428] Setting key 1cf6ce548413d7cea9d7843c6f5e69af
I20260925 14:53:35.298501 21364 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc17b409427d4eb2af7754b3f2502901" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38419 } }
I20260925 14:53:35.298691 21364 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:35.298732 21364 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc17b409427d4eb2af7754b3f2502901, State: Initialized, Role: FOLLOWER
I20260925 14:53:35.298843 21364 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc17b409427d4eb2af7754b3f2502901" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38419 } }
I20260925 14:53:35.298931 21364 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:35.298972 21364 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:35.299019 21364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:35.300019 21364 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc17b409427d4eb2af7754b3f2502901" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38419 } }
I20260925 14:53:35.300127 21364 leader_election.cc:302] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc17b409427d4eb2af7754b3f2502901; no voters:
I20260925 14:53:35.300276 21364 leader_election.cc:288] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:35.300323 21369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:35.300452 21369 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [term 1 LEADER]: Becoming Leader. State: Replica: fc17b409427d4eb2af7754b3f2502901, State: Running, Role: LEADER
I20260925 14:53:35.300522 21364 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:35.300534 21369 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc17b409427d4eb2af7754b3f2502901" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38419 } }
I20260925 14:53:35.300877 21369 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc17b409427d4eb2af7754b3f2502901. Latest consensus state: current_term: 1 leader_uuid: "fc17b409427d4eb2af7754b3f2502901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc17b409427d4eb2af7754b3f2502901" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38419 } } }
I20260925 14:53:35.300962 21369 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:35.300882 21370 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc17b409427d4eb2af7754b3f2502901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc17b409427d4eb2af7754b3f2502901" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38419 } } }
I20260925 14:53:35.301160 21370 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:35.301283 21375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:35.301985 21375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:35.303596 21375 catalog_manager.cc:1347] Generated new cluster ID: 6793a432dc1a45c39a10ce967e8adebc
I20260925 14:53:35.303645 21375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:35.313033 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.298786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.20.199.62:38419)
0925 14:53:35.299131 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:35.299137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:35.299208 (+ 71us) server_negotiation.cc:407] Connection header received
0925 14:53:35.301751 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:35.301759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:35.301820 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:35.301934 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:35.303848 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.304788 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.305869 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.306277 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.306412 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.309566 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:35.309570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:35.309570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:35.309579 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:35.311180 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.311593 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.311595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.311596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.311713 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.311990 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.311991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.311991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.312231 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:35.312396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:35.312647 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:53:35.312777 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":83,"threads_started":1}
I20260925 14:53:35.325906 21375 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:35.326411 21375 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:35.331638 21375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc17b409427d4eb2af7754b3f2502901: Generated new TSK 0
I20260925 14:53:35.331820 21375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:35.360566 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38419
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:35.458575 21390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:35.458781 21390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:35.458819 21390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:35.459981 21390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:35.460043 21390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:35.460086 21390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:35.461557 21390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:35.461808 21390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:35.462020 21390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:35.466760 21396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:35.466784 21395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:35.466790 21398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:35.466933 21390 server_base.cc:1034] running on GCE node
I20260925 14:53:35.467165 21390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:35.467401 21390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:35.468559 21390 hybrid_clock.cc:648] HybridClock initialized: now 1790348015468550 us; error 38 us; skew 500 ppm
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:35.471097 21390 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:35.471529 21390 webserver.cc:460] Webserver started at http://127.20.199.1:36987/ using document root <none> and password file <none>
I20260925 14:53:35.471812 21390 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:35.471856 21390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:35.471997 21390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:35.473024 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "63935f9bfa2546ba9ed45e2f19ebf297"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "621bab377b1ab8a5665baeb5adae249f"
server_key_iv: "b468d25c07f6da864a2e1edd96109e75"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.473366 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "63935f9bfa2546ba9ed45e2f19ebf297"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "621bab377b1ab8a5665baeb5adae249f"
server_key_iv: "b468d25c07f6da864a2e1edd96109e75"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.474669 21390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:35.475343 21405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.475509 21390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:35.475605 21390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "63935f9bfa2546ba9ed45e2f19ebf297"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "621bab377b1ab8a5665baeb5adae249f"
server_key_iv: "b468d25c07f6da864a2e1edd96109e75"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.475680 21390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:35.501386 21390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:35.503255 21390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:35.503413 21390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:35.503636 21390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:35.503949 21390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:35.504004 21390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.504045 21390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:35.504071 21390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.509852 21390 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:36013
I20260925 14:53:35.509903 21518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:36013 every 8 connection(s)
I20260925 14:53:35.510200 21390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:35.515707 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 21390
I20260925 14:53:35.515777 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:35.515930 21276 external_mini_cluster.cc:1428] Setting key 4831811d5130928f4c71849f87840eb5
I20260925 14:53:35.519416 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.510838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38419 (local address 127.20.199.1:54129)
0925 14:53:35.511010 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:35.511051 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:35.511540 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:35.511755 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:35.511758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:35.512302 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:35.512486 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:35.512491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.513127 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.513128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:35.513989 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:35.513992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.514337 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.514338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:35.514433 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.514837 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:35.514845 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:35.516353 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:35.518155 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:35.518161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:35.518163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:35.518367 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:35.518587 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:35.518587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:35.518588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:35.518678 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:35.518952 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:35.518956 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:35.519060 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 14:53:35.519142 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:53:35.519255 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260925 14:53:35.519431 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.510851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:54129 (local address 127.20.199.62:38419)
0925 14:53:35.510920 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:35.510927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:35.511143 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:53:35.511636 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:35.511637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:35.511647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:35.511697 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:35.512564 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.513045 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.514068 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.514268 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.514322 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.516433 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:35.516434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:35.516434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:35.516440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:35.518062 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.518438 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.518439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.518439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.518512 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.518740 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.518741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.518742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.518889 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:35.518958 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:35.519190 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:53:35.519299 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:35.519930 21519 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38419
I20260925 14:53:35.520025 21519 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:35.520190 21519 heartbeater.cc:511] Master 127.20.199.62:38419 requested a full tablet report, sending...
I20260925 14:53:35.520697 21328 ts_manager.cc:194] Registered new tserver with Master: 63935f9bfa2546ba9ed45e2f19ebf297 (127.20.199.1:36013)
I20260925 14:53:35.521374 21328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:54129
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:35.540640 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38419
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:35.619590 21524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:35.619777 21524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:35.619800 21524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:35.620923 21524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:35.620963 21524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:35.620990 21524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:35.622478 21524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:35.622717 21524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:35.622939 21524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:35.627846 21530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:35.627842 21529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:35.627988 21524 server_base.cc:1034] running on GCE node
W20260925 14:53:35.627839 21532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:35.628285 21524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:35.628588 21524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:35.629746 21524 hybrid_clock.cc:648] HybridClock initialized: now 1790348015629735 us; error 39 us; skew 500 ppm
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:35.632141 21524 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:35.632490 21524 webserver.cc:460] Webserver started at http://127.20.199.2:37783/ using document root <none> and password file <none>
I20260925 14:53:35.632664 21524 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:35.632702 21524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:35.632786 21524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:35.633790 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "746e6c4ba92a4d24bdf39b55af6a0851"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "583a76a22f782a96cef039684cc98dca"
server_key_iv: "3f084b4034e2b79b889f352cb7417b8b"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.634074 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "746e6c4ba92a4d24bdf39b55af6a0851"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "583a76a22f782a96cef039684cc98dca"
server_key_iv: "3f084b4034e2b79b889f352cb7417b8b"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.635334 21524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:35.636008 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.636237 21524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:35.636315 21524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "746e6c4ba92a4d24bdf39b55af6a0851"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "583a76a22f782a96cef039684cc98dca"
server_key_iv: "3f084b4034e2b79b889f352cb7417b8b"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.636385 21524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:35.650529 21524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:35.652704 21524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:35.652809 21524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:35.652973 21524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:35.653199 21524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:35.653227 21524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.653247 21524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:35.653260 21524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.659037 21524 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:40977
I20260925 14:53:35.659121 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:40977 every 8 connection(s)
I20260925 14:53:35.659392 21524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:35.665011 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 21524
I20260925 14:53:35.665133 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:35.665292 21276 external_mini_cluster.cc:1428] Setting key 72105c88055200bce4da134266e3a7e0
I20260925 14:53:35.668329 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.660118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57635 (local address 127.20.199.62:38419)
0925 14:53:35.660206 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:35.660208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:35.660354 (+ 146us) server_negotiation.cc:407] Connection header received
0925 14:53:35.660777 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:35.660778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:35.660787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:35.660836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:35.661742 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.662245 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.663192 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.663368 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.663416 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.665554 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:35.665555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:35.665556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:35.665563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:35.666956 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.667392 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.667393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.667394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.667454 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.667701 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.667702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.667703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.667814 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:35.667883 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:35.668152 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:53:35.668233 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:53:35.668498 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.660069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38419 (local address 127.20.199.2:57635)
0925 14:53:35.660238 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:35.660270 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:35.660702 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:35.660885 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:35.660888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:35.661458 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:35.661672 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:35.661677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.662319 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.662319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:35.663107 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:35.663109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.663441 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.663442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:35.663539 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.664057 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:35.664065 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:35.665454 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:35.667076 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:35.667082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:35.667084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:35.667301 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:35.667521 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:35.667521 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:35.667522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:35.667633 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:35.667887 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:35.667893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:35.668022 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 14:53:35.668119 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:53:35.668268 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":109,"threads_started":1}
I20260925 14:53:35.669147 21653 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38419
I20260925 14:53:35.669222 21653 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:35.669441 21653 heartbeater.cc:511] Master 127.20.199.62:38419 requested a full tablet report, sending...
I20260925 14:53:35.669821 21328 ts_manager.cc:194] Registered new tserver with Master: 746e6c4ba92a4d24bdf39b55af6a0851 (127.20.199.2:40977)
I20260925 14:53:35.670208 21328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:57635
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:35.689715 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38419
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:35.768693 21658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:35.768873 21658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:35.768895 21658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:35.770036 21658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:35.770080 21658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:35.770108 21658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:35.771538 21658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:35.771725 21658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:35.771888 21658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:35.776690 21663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:35.776711 21664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:35.776731 21658 server_base.cc:1034] running on GCE node
W20260925 14:53:35.776713 21666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:35.777055 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:35.777313 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:35.778461 21658 hybrid_clock.cc:648] HybridClock initialized: now 1790348015778456 us; error 42 us; skew 500 ppm
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:35.780789 21658 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:35.781215 21658 webserver.cc:460] Webserver started at http://127.20.199.3:45253/ using document root <none> and password file <none>
I20260925 14:53:35.781389 21658 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:35.781491 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:35.781580 21658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:35.782464 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "5f6526634f5647f2bc8f73f9e45be19d"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "5e898fa2c8a34078e4554adb9b1e2526"
server_key_iv: "75d38f55159e86dc423d441e79a9a210"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.782761 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "5f6526634f5647f2bc8f73f9e45be19d"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "5e898fa2c8a34078e4554adb9b1e2526"
server_key_iv: "75d38f55159e86dc423d441e79a9a210"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.783989 21658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:53:35.784639 21673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.784884 21658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:35.784953 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "5f6526634f5647f2bc8f73f9e45be19d"
format_stamp: "Formatted at 2026-09-25 14:53:35 on dist-test-slave-8qt6"
server_key: "5e898fa2c8a34078e4554adb9b1e2526"
server_key_iv: "75d38f55159e86dc423d441e79a9a210"
server_key_version: "encryptionkey@0"
I20260925 14:53:35.785034 21658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:35.795405 21658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:35.797212 21658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:35.797361 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:35.797626 21658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:35.797914 21658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:35.797947 21658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.797968 21658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:35.798000 21658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:35.804284 21658 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:37331
I20260925 14:53:35.804325 21786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:37331 every 8 connection(s)
I20260925 14:53:35.804621 21658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:35.813071 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.805287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:58855 (local address 127.20.199.62:38419)
0925 14:53:35.805329 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:35.805331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:35.805546 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:53:35.806082 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:35.806084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:35.806102 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:35.806159 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:35.807070 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.807568 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.808565 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.808758 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.808841 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.810866 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:35.810867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:35.810868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:35.810874 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:35.812005 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.812343 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.812344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.812344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.812392 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.812597 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.812598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.812598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.812693 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:35.812741 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:35.812954 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 14:53:35.813011 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:53:35.813305 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.805255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38419 (local address 127.20.199.3:58855)
0925 14:53:35.805384 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:35.805452 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:35.805969 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:35.806227 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:35.806230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:35.806784 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:35.806974 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:35.806978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.807671 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.807672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:35.808468 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:35.808470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.808849 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.808850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:35.808983 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.809492 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:35.809506 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:35.810780 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:35.812074 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:35.812079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:35.812080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:35.812280 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:35.812450 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:35.812451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:35.812451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:35.812540 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:35.812745 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:35.812748 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:35.812849 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 14:53:35.812926 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:35.813040 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260925 14:53:35.813870 21787 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38419
I20260925 14:53:35.813910 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 21658
I20260925 14:53:35.813999 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:35.814002 21787 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:35.814154 21276 external_mini_cluster.cc:1428] Setting key 74a3a588e2896a52ce7f60f1b1340f0c
I20260925 14:53:35.814164 21787 heartbeater.cc:511] Master 127.20.199.62:38419 requested a full tablet report, sending...
I20260925 14:53:35.814548 21328 ts_manager.cc:194] Registered new tserver with Master: 5f6526634f5647f2bc8f73f9e45be19d (127.20.199.3:37331)
I20260925 14:53:35.815006 21328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:58855
I20260925 14:53:35.815331 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:35.837450 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.829930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.20.199.1:36013)
0925 14:53:35.830095 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:35.830098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:35.830108 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:35.830131 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:35.830133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:35.830146 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:35.830228 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:35.830761 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.831461 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.832257 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.832548 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.832609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.834701 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:35.834713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:35.834713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:35.834719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:35.836240 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.836608 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.836612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.836612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.836671 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.836877 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.836878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.836878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.837006 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:35.837168 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:35.837240 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:53:35.837314 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260925 14:53:35.837796 21433 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33194
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:35.858819 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:35.851813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.20.199.1:36013)
0925 14:53:35.851879 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:35.851881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:35.851892 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:35.851970 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:35.851971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:35.851981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:35.852029 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:35.852562 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.853039 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.853798 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:35.854038 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:35.854108 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:35.856300 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:35.856301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:35.856301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:35.856309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:35.857598 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.857987 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.857988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.857988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.858052 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:35.858268 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:35.858269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:35.858270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:35.858383 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:35.858517 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:35.858620 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:53:35.858704 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:53:35.859411 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 21390
I20260925 14:53:35.864255 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 21524
I20260925 14:53:35.868259 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 21658
I20260925 14:53:35.872180 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 21296
2026-09-25T14:53:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1845 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21801](info): set up 2 sockets
Sep 25 14:53:35 dist-test-slave-8qt6 krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260925 14:53:36.904860 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:36 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:37.020166 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:39723
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:39723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:37.101154 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:37.101361 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:37.101382 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:37.102504 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:37.102569 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:37.102586 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:37.102628 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:37.102659 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:37.104241 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:39723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:39723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:37.104485 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:37.104707 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:37.109561 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:37.109550 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:37.109563 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:37.109838 21814 server_base.cc:1034] running on GCE node
I20260925 14:53:37.110076 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:37.110296 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:37.111431 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790348017111420 us; error 27 us; skew 500 ppm
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:37.113710 21814 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:37.114040 21814 webserver.cc:460] Webserver started at http://127.20.199.62:39631/ using document root <none> and password file <none>
I20260925 14:53:37.114255 21814 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:37.114296 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:37.114379 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:37.115291 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "18f62a9458884f1fbf9670d289ca9bbb"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "738244800793954a407f2393fcb53cb1"
server_key_iv: "ba9ef98b655398dd30d3d60a546679a7"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.115577 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "18f62a9458884f1fbf9670d289ca9bbb"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "738244800793954a407f2393fcb53cb1"
server_key_iv: "ba9ef98b655398dd30d3d60a546679a7"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.116756 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:37.117377 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.117569 21814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:37.117664 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "18f62a9458884f1fbf9670d289ca9bbb"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "738244800793954a407f2393fcb53cb1"
server_key_iv: "ba9ef98b655398dd30d3d60a546679a7"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.117744 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:37.133500 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:37.135268 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:37.135424 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:37.139287 21814 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:39723
I20260925 14:53:37.139333 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:39723 every 8 connection(s)
I20260925 14:53:37.139694 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:37.140543 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:37.142884 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb: Bootstrap starting.
I20260925 14:53:37.143455 21882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:37.143747 21882 log.cc:824] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:37.144462 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb: No bootstrap required, opened a new log
I20260925 14:53:37.144608 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 21814
I20260925 14:53:37.144676 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:37.144785 21276 external_mini_cluster.cc:1428] Setting key 59a86eaa2db9bf606a5509b9d69f169b
I20260925 14:53:37.146324 21882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f62a9458884f1fbf9670d289ca9bbb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39723 } }
I20260925 14:53:37.146483 21882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:37.146530 21882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f62a9458884f1fbf9670d289ca9bbb, State: Initialized, Role: FOLLOWER
I20260925 14:53:37.146692 21882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f62a9458884f1fbf9670d289ca9bbb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39723 } }
I20260925 14:53:37.146785 21882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:37.146836 21882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:37.146898 21882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:37.147758 21882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f62a9458884f1fbf9670d289ca9bbb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39723 } }
I20260925 14:53:37.147882 21882 leader_election.cc:302] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18f62a9458884f1fbf9670d289ca9bbb; no voters:
I20260925 14:53:37.148092 21882 leader_election.cc:288] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:37.148114 21887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:37.148231 21887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [term 1 LEADER]: Becoming Leader. State: Replica: 18f62a9458884f1fbf9670d289ca9bbb, State: Running, Role: LEADER
I20260925 14:53:37.148324 21887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f62a9458884f1fbf9670d289ca9bbb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39723 } }
I20260925 14:53:37.148360 21882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:37.148669 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18f62a9458884f1fbf9670d289ca9bbb. Latest consensus state: current_term: 1 leader_uuid: "18f62a9458884f1fbf9670d289ca9bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f62a9458884f1fbf9670d289ca9bbb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39723 } } }
I20260925 14:53:37.148761 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:37.148895 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18f62a9458884f1fbf9670d289ca9bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f62a9458884f1fbf9670d289ca9bbb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39723 } } }
I20260925 14:53:37.148942 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:37.149047 21892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:37.149534 21892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:37.151178 21892 catalog_manager.cc:1347] Generated new cluster ID: 42ecf3b5bea143318e266d622bc4199a
I20260925 14:53:37.151235 21892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:37.155719 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.145229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.20.199.62:39723)
0925 14:53:37.145442 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.145452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.145479 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:53:37.146298 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.146304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.146346 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.146480 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.147188 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.148131 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.149186 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.149901 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.150002 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.152571 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.152575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.152575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.152594 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.154181 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.154550 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.154553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.154553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.154659 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.154869 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.154870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.154871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.155088 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.155244 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.155347 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.155576 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260925 14:53:37.164974 21892 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:37.165514 21892 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:37.175443 21892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18f62a9458884f1fbf9670d289ca9bbb: Generated new TSK 0
I20260925 14:53:37.175637 21892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:37.209064 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:39723
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:37.289259 21908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:37.289466 21908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:37.289496 21908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:37.290973 21908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:37.291018 21908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:37.291044 21908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:37.292441 21908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:39723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:37.292624 21908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:37.292794 21908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:37.297771 21908 server_base.cc:1034] running on GCE node
W20260925 14:53:37.297710 21914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:37.297858 21913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:37.297722 21916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:37.298188 21908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:37.298432 21908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:37.299600 21908 hybrid_clock.cc:648] HybridClock initialized: now 1790348017299577 us; error 58 us; skew 500 ppm
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:37.302073 21908 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:37.302505 21908 webserver.cc:460] Webserver started at http://127.20.199.1:33955/ using document root <none> and password file <none>
I20260925 14:53:37.302682 21908 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:37.302721 21908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:37.302805 21908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:37.303822 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "361e21b2071948f68d0110f777e7f242"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "0b09fce165afec8c09643dcdd425f7f4"
server_key_iv: "6a0ffabcde1f3bf2de18f2a2fe128759"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.304123 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "361e21b2071948f68d0110f777e7f242"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "0b09fce165afec8c09643dcdd425f7f4"
server_key_iv: "6a0ffabcde1f3bf2de18f2a2fe128759"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.305325 21908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:53:37.306105 21923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.306296 21908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:37.306389 21908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "361e21b2071948f68d0110f777e7f242"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "0b09fce165afec8c09643dcdd425f7f4"
server_key_iv: "6a0ffabcde1f3bf2de18f2a2fe128759"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.306442 21908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:37.321055 21908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:37.323167 21908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:37.323336 21908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:37.323633 21908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:37.323959 21908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:37.323994 21908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.324018 21908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:37.324030 21908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.329661 21908 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:41235
I20260925 14:53:37.329716 22036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:41235 every 8 connection(s)
I20260925 14:53:37.330062 21908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:37.333679 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 21908
I20260925 14:53:37.333770 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:37.333921 21276 external_mini_cluster.cc:1428] Setting key 2123d6cb4f85c6a6234e17e7fe0fddde
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:37.340235 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.330787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:54935 (local address 127.20.199.62:39723)
0925 14:53:37.330850 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.330853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.331098 (+ 245us) server_negotiation.cc:407] Connection header received
0925 14:53:37.331988 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.331989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.332003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.332057 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.332619 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.333114 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.334467 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.334722 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.334813 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.337267 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.337268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.337268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.337275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.338729 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.339270 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.339271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.339271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.339322 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.339583 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.339584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.339584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.339678 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.339726 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.340096 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.340171 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:53:37.340463 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.330772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:39723 (local address 127.20.199.1:54935)
0925 14:53:37.330952 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.331004 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.331892 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.332116 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.332119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.332358 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:37.332540 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.332544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.333189 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.333190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.334387 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.334390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.334828 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.334829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.334970 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.335490 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:37.335498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:37.337163 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:37.338823 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:37.338830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:37.338832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:37.339182 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:37.339380 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:37.339380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:37.339381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:37.339517 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:37.339741 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:37.339747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:37.339917 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.340054 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.340218 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 14:53:37.340951 22037 heartbeater.cc:347] Connected to a master server at 127.20.199.62:39723
I20260925 14:53:37.341056 22037 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:37.341265 22037 heartbeater.cc:511] Master 127.20.199.62:39723 requested a full tablet report, sending...
I20260925 14:53:37.341832 21846 ts_manager.cc:194] Registered new tserver with Master: 361e21b2071948f68d0110f777e7f242 (127.20.199.1:41235)
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
I20260925 14:53:37.342480 21846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:54935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:37.359956 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:39723
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:37.440447 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:37.440696 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:37.440735 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:37.441898 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:37.441972 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:37.442020 22042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:37.443508 22042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:39723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:37.443789 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:37.444036 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:37.448967 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:37.448979 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:37.449185 22042 server_base.cc:1034] running on GCE node
W20260925 14:53:37.449237 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:37.449469 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:37.449743 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:37.450946 22042 hybrid_clock.cc:648] HybridClock initialized: now 1790348017450932 us; error 25 us; skew 500 ppm
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:37.453341 22042 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:37.453783 22042 webserver.cc:460] Webserver started at http://127.20.199.2:43107/ using document root <none> and password file <none>
I20260925 14:53:37.454047 22042 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:37.454115 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:37.454273 22042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:37.455320 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "a194f8b4b3144a26a1fd2695c28081f9"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "4a09a24b80682aaee61024260057a1a5"
server_key_iv: "51c9f6bc3ffe10f7305466ac58911f35"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.455679 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "a194f8b4b3144a26a1fd2695c28081f9"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "4a09a24b80682aaee61024260057a1a5"
server_key_iv: "51c9f6bc3ffe10f7305466ac58911f35"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.457031 22042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:37.457784 22057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.457947 22042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.458045 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "a194f8b4b3144a26a1fd2695c28081f9"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "4a09a24b80682aaee61024260057a1a5"
server_key_iv: "51c9f6bc3ffe10f7305466ac58911f35"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.458122 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:37.482228 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:37.484086 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:37.484249 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:37.484500 22042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:37.484748 22042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:37.484777 22042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.484799 22042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:37.484812 22042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.490442 22042 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:39377
I20260925 14:53:37.490482 22170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:39377 every 8 connection(s)
I20260925 14:53:37.490782 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:37.494529 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22042
I20260925 14:53:37.494623 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:37.494819 21276 external_mini_cluster.cc:1428] Setting key 60238861aa420084cc3a0e0c2a7d8b8f
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:37.500365 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.491456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:41101 (local address 127.20.199.62:39723)
0925 14:53:37.491543 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.491545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.491749 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:53:37.492690 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.492691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.492702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.492757 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.493373 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.493959 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.495089 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.495301 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.495384 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.497564 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.497566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.497566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.497574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.498972 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.499398 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.499399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.499399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.499459 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.499714 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.499715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.499716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.499834 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.499911 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.500192 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.500276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:53:37.500568 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.491442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:39723 (local address 127.20.199.2:41101)
0925 14:53:37.491630 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.491663 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.492575 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.492833 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.492836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.493081 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:37.493267 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.493272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.494058 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.494059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.495002 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.495004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.495394 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.495395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.495529 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.496065 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:37.496073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:37.497466 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:37.499073 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:37.499080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:37.499083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:37.499307 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:37.499532 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:37.499533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:37.499533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:37.499647 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:37.499901 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:37.499907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:37.500056 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.500159 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.500318 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260925 14:53:37.501047 22171 heartbeater.cc:347] Connected to a master server at 127.20.199.62:39723
I20260925 14:53:37.501140 22171 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:37.501338 22171 heartbeater.cc:511] Master 127.20.199.62:39723 requested a full tablet report, sending...
I20260925 14:53:37.501780 21846 ts_manager.cc:194] Registered new tserver with Master: a194f8b4b3144a26a1fd2695c28081f9 (127.20.199.2:39377)
I20260925 14:53:37.502166 21846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:41101
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:37.520766 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:39723
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:37.601197 22176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:37.601420 22176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:37.601444 22176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:37.602528 22176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:37.602568 22176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:37.602594 22176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:37.604009 22176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:39723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:37.604194 22176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:37.604369 22176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:37.609381 22181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:37.609545 22182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:37.609622 22176 server_base.cc:1034] running on GCE node
W20260925 14:53:37.609393 22184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:37.609933 22176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:37.610173 22176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:37.611335 22176 hybrid_clock.cc:648] HybridClock initialized: now 1790348017611324 us; error 28 us; skew 500 ppm
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:37.613878 22176 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:37.614367 22176 webserver.cc:460] Webserver started at http://127.20.199.3:43327/ using document root <none> and password file <none>
I20260925 14:53:37.614589 22176 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:37.614655 22176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:37.614804 22176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:37.615869 22176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "dfab05e879694ff49b458e2f75252473"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "d5e9299816a32f437729b1bcc2e57489"
server_key_iv: "da1267f78c35f91d853bd0f435d374e9"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.616232 22176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "dfab05e879694ff49b458e2f75252473"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "d5e9299816a32f437729b1bcc2e57489"
server_key_iv: "da1267f78c35f91d853bd0f435d374e9"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.617677 22176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:53:37.618391 22191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.618556 22176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:37.618659 22176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "dfab05e879694ff49b458e2f75252473"
format_stamp: "Formatted at 2026-09-25 14:53:37 on dist-test-slave-8qt6"
server_key: "d5e9299816a32f437729b1bcc2e57489"
server_key_iv: "da1267f78c35f91d853bd0f435d374e9"
server_key_version: "encryptionkey@0"
I20260925 14:53:37.618736 22176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:37.644827 22176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:37.647019 22176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:37.647189 22176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:37.647446 22176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:37.647763 22176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:37.647816 22176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.647851 22176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:37.647899 22176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:37.654201 22176 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:43863
I20260925 14:53:37.654258 22304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:43863 every 8 connection(s)
I20260925 14:53:37.654609 22176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:37.655776 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22176
I20260925 14:53:37.655865 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:37.656014 21276 external_mini_cluster.cc:1428] Setting key ffc303b23c8905695d039b96e8cf5ea3
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:37.663790 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.655322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:55933 (local address 127.20.199.62:39723)
0925 14:53:37.655386 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.655389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.655595 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:53:37.656889 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.656890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.656899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.656930 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.657584 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.658062 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.659012 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.659185 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.659234 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.661301 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.661302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.661302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.661308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.662723 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.663074 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.663075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.663075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.663122 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.663321 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.663321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.663321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.663414 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.663462 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.663676 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.663730 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:53:37.663993 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.655316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:39723 (local address 127.20.199.3:55933)
0925 14:53:37.655468 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.655504 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.656794 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.656996 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.656999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.657249 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:37.657512 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.657517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.658137 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.658138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.658946 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.658948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.659253 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.659254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.659350 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.659787 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:37.659795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:37.661221 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:37.662795 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:37.662801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:37.662802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:37.663009 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:37.663176 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:37.663177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:37.663177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:37.663268 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:37.663465 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:37.663469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:37.663572 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.663651 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.663760 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260925 14:53:37.664445 22305 heartbeater.cc:347] Connected to a master server at 127.20.199.62:39723
I20260925 14:53:37.664572 22305 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:37.664752 22305 heartbeater.cc:511] Master 127.20.199.62:39723 requested a full tablet report, sending...
I20260925 14:53:37.665066 21846 ts_manager.cc:194] Registered new tserver with Master: dfab05e879694ff49b458e2f75252473 (127.20.199.3:43863)
I20260925 14:53:37.665474 21846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:55933
I20260925 14:53:37.667636 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:37.674557 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.669176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.20.199.62:39723)
0925 14:53:37.669242 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.669243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.669292 (+ 49us) server_negotiation.cc:407] Connection header received
0925 14:53:37.669339 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.669340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.669347 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.669391 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.669957 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.670392 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.671057 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.671223 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.671269 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.672271 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.672272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.672272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.672277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.673643 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.673941 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.673942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.673942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.673996 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.674152 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.674152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.674152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.674272 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.674340 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.674428 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.674478 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:53:37.676584 21846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:37.678138 21846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:37.684579 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.680397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.20.199.2:39377)
0925 14:53:37.680973 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.680978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.680988 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:37.681031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.681033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.681048 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.681132 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:37.681367 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.682421 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.683462 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.684179 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.684258 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.684319 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.684453 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.684495 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":91,"threads_started":1}
I20260925 14:53:37.684582 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.680277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:39377 (local address 127.0.0.1:49008)
0925 14:53:37.680395 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.680430 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.680494 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.681267 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.681268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.681269 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:37.681295 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.681296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.682518 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.682519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.683366 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.683368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.684261 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.684262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.684338 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.684354 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.684426 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.684486 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260925 14:53:37.684770 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.680548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.20.199.3:43863)
0925 14:53:37.680701 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.680705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.680713 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:37.680742 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.680743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.680755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.680841 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:37.680974 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.682179 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.683624 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.684282 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.684346 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.684394 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.684627 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.684670 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260925 14:53:37.684582 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.680231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:43863 (local address 127.0.0.1:39428)
0925 14:53:37.680388 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.680430 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.680497 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.680846 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.680849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.680852 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:37.680917 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.680919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.682278 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.682278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.683552 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.683554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.684351 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.684352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.684409 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.684412 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.684461 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.684490 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":49,"thread_start_us":42,"threads_started":1}
I20260925 14:53:37.685534 22105 tablet_service.cc:1467] Processing CreateTablet for tablet ec54c2bf385b483babb6ae68f8165595 (DEFAULT_TABLE table=test-table [id=c6d7ae7bc30c4c4d99b91f5787d49609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:37.685762 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.680429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:41235 (local address 127.0.0.1:46968)
0925 14:53:37.680785 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.680797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.680851 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.681143 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.681144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.681146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:37.681191 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.681193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.682132 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.682133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.683539 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.683543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.685513 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.685514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.685575 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.685580 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.685641 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.685683 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":79,"thread_start_us":58,"threads_started":1}
I20260925 14:53:37.685770 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.680782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.20.199.1:41235)
0925 14:53:37.680988 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.680992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.681001 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:37.681035 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.681037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.681050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.681132 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:37.681251 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.682012 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.684453 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.685350 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.685496 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.685558 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.685651 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.685690 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260925 14:53:37.685721 22239 tablet_service.cc:1467] Processing CreateTablet for tablet ec54c2bf385b483babb6ae68f8165595 (DEFAULT_TABLE table=test-table [id=c6d7ae7bc30c4c4d99b91f5787d49609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:37.685905 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54c2bf385b483babb6ae68f8165595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:37.686081 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54c2bf385b483babb6ae68f8165595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:37.686398 21971 tablet_service.cc:1467] Processing CreateTablet for tablet ec54c2bf385b483babb6ae68f8165595 (DEFAULT_TABLE table=test-table [id=c6d7ae7bc30c4c4d99b91f5787d49609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:37.686674 21971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54c2bf385b483babb6ae68f8165595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:37.688272 22322 tablet_bootstrap.cc:492] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: Bootstrap starting.
I20260925 14:53:37.688922 22322 tablet_bootstrap.cc:654] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:37.689334 22323 tablet_bootstrap.cc:492] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473: Bootstrap starting.
I20260925 14:53:37.689458 22322 log.cc:824] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:37.690021 22323 tablet_bootstrap.cc:654] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:37.690004 22325 tablet_bootstrap.cc:492] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: Bootstrap starting.
I20260925 14:53:37.690348 22322 tablet_bootstrap.cc:492] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: No bootstrap required, opened a new log
I20260925 14:53:37.690428 22322 ts_tablet_manager.cc:1397] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:37.690442 22323 log.cc:824] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:37.690878 22325 tablet_bootstrap.cc:654] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:37.691030 22323 tablet_bootstrap.cc:492] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473: No bootstrap required, opened a new log
I20260925 14:53:37.691080 22323 ts_tablet_manager.cc:1397] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:37.691547 22325 log.cc:824] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:37.692332 22325 tablet_bootstrap.cc:492] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: No bootstrap required, opened a new log
I20260925 14:53:37.692445 22325 ts_tablet_manager.cc:1397] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:37.692626 22322 raft_consensus.cc:348] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.692800 22322 raft_consensus.cc:374] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:37.692703 22323 raft_consensus.cc:348] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.692844 22322 raft_consensus.cc:729] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a194f8b4b3144a26a1fd2695c28081f9, State: Initialized, Role: FOLLOWER
I20260925 14:53:37.692864 22323 raft_consensus.cc:374] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:37.692891 22323 raft_consensus.cc:729] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfab05e879694ff49b458e2f75252473, State: Initialized, Role: FOLLOWER
I20260925 14:53:37.692956 22322 consensus_queue.cc:260] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.693013 22323 consensus_queue.cc:260] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.693244 22322 ts_tablet_manager.cc:1428] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:37.693271 22323 ts_tablet_manager.cc:1428] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:37.693272 22171 heartbeater.cc:503] Master 127.20.199.62:39723 was elected leader, sending a full tablet report...
I20260925 14:53:37.693390 22305 heartbeater.cc:503] Master 127.20.199.62:39723 was elected leader, sending a full tablet report...
I20260925 14:53:37.694787 22325 raft_consensus.cc:348] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.694916 22325 raft_consensus.cc:374] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:37.694962 22325 raft_consensus.cc:729] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361e21b2071948f68d0110f777e7f242, State: Initialized, Role: FOLLOWER
I20260925 14:53:37.695070 22325 consensus_queue.cc:260] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.695325 22325 ts_tablet_manager.cc:1428] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:53:37.695371 22037 heartbeater.cc:503] Master 127.20.199.62:39723 was elected leader, sending a full tablet report...
W20260925 14:53:37.741336 22172 tablet.cc:2369] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:37.767589 22328 raft_consensus.cc:482] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:37.767755 22328 raft_consensus.cc:504] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.768193 22328 leader_election.cc:288] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfab05e879694ff49b458e2f75252473 (127.20.199.3:43863), 361e21b2071948f68d0110f777e7f242 (127.20.199.1:41235)
I20260925 14:53:37.771229 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.768379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:43863 (local address 127.20.199.2:41027)
0925 14:53:37.768439 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.768452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.768530 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.768713 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.768714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.768716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:37.768800 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.768802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.769518 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.769519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.770306 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.770308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.770945 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.770946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.771054 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.771059 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.771115 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.771159 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:53:37.771260 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.768457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:41027 (local address 127.20.199.3:43863)
0925 14:53:37.768526 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.768529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.768540 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:37.768600 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.768601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.768616 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.768670 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:37.768901 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.769445 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.770402 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.770813 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.770868 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.770874 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.771142 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.771182 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:53:37.771425 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.768485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:41235 (local address 127.20.199.2:56691)
0925 14:53:37.768649 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:37.768658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:37.768709 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:37.768875 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:37.768876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:37.768877 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:37.768948 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.768950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.769641 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.769641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.770408 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:37.770410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.771147 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.771148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:37.771218 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.771222 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:37.771294 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:53:37.771334 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260925 14:53:37.771443 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.768570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:56691 (local address 127.20.199.1:41235)
0925 14:53:37.768616 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.768618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.768725 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:53:37.768771 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.768772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.768787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.768842 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:37.769012 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.769558 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.770495 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.771040 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.771126 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.771135 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.771307 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.771365 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:37.771629 22259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54c2bf385b483babb6ae68f8165595" candidate_uuid: "a194f8b4b3144a26a1fd2695c28081f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfab05e879694ff49b458e2f75252473" is_pre_election: true
I20260925 14:53:37.771768 22259 raft_consensus.cc:2454] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a194f8b4b3144a26a1fd2695c28081f9 in term 0.
I20260925 14:53:37.771950 22058 leader_election.cc:302] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a194f8b4b3144a26a1fd2695c28081f9, dfab05e879694ff49b458e2f75252473; no voters:
I20260925 14:53:37.772071 22328 raft_consensus.cc:2781] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:37.772152 22328 raft_consensus.cc:482] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:37.772181 22328 raft_consensus.cc:3037] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:37.772286 21991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54c2bf385b483babb6ae68f8165595" candidate_uuid: "a194f8b4b3144a26a1fd2695c28081f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361e21b2071948f68d0110f777e7f242" is_pre_election: true
I20260925 14:53:37.772446 21991 raft_consensus.cc:2454] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a194f8b4b3144a26a1fd2695c28081f9 in term 0.
I20260925 14:53:37.773005 22328 raft_consensus.cc:504] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.773138 22328 leader_election.cc:288] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [CANDIDATE]: Term 1 election: Requested vote from peers dfab05e879694ff49b458e2f75252473 (127.20.199.3:43863), 361e21b2071948f68d0110f777e7f242 (127.20.199.1:41235)
I20260925 14:53:37.773427 21991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54c2bf385b483babb6ae68f8165595" candidate_uuid: "a194f8b4b3144a26a1fd2695c28081f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361e21b2071948f68d0110f777e7f242"
I20260925 14:53:37.773419 22259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54c2bf385b483babb6ae68f8165595" candidate_uuid: "a194f8b4b3144a26a1fd2695c28081f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfab05e879694ff49b458e2f75252473"
I20260925 14:53:37.773501 22259 raft_consensus.cc:3037] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:37.773514 21991 raft_consensus.cc:3037] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:37.774379 22259 raft_consensus.cc:2454] T ec54c2bf385b483babb6ae68f8165595 P dfab05e879694ff49b458e2f75252473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a194f8b4b3144a26a1fd2695c28081f9 in term 1.
I20260925 14:53:37.774381 21991 raft_consensus.cc:2454] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a194f8b4b3144a26a1fd2695c28081f9 in term 1.
I20260925 14:53:37.774544 22058 leader_election.cc:302] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361e21b2071948f68d0110f777e7f242, a194f8b4b3144a26a1fd2695c28081f9; no voters:
I20260925 14:53:37.774650 22328 raft_consensus.cc:2781] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:37.774813 22328 raft_consensus.cc:686] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [term 1 LEADER]: Becoming Leader. State: Replica: a194f8b4b3144a26a1fd2695c28081f9, State: Running, Role: LEADER
I20260925 14:53:37.774946 22328 consensus_queue.cc:237] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } }
I20260925 14:53:37.775770 21845 catalog_manager.cc:5495] T ec54c2bf385b483babb6ae68f8165595 P a194f8b4b3144a26a1fd2695c28081f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a194f8b4b3144a26a1fd2695c28081f9 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "a194f8b4b3144a26a1fd2695c28081f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfab05e879694ff49b458e2f75252473" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "361e21b2071948f68d0110f777e7f242" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a194f8b4b3144a26a1fd2695c28081f9" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39377 } health_report { overall_health: HEALTHY } } }
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:37.806342 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.798781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.20.199.1:41235)
0925 14:53:37.798877 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.798880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.798963 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:53:37.799031 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.799032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.799042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.799116 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.799680 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.800158 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.800869 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.801066 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.801141 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.803447 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.803468 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.803468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.803476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.805096 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.805514 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.805518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.805518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.805579 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.805794 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.805796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.805796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.805949 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.806112 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.806151 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.806234 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:37.829792 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.822726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.20.199.1:41235)
0925 14:53:37.822798 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.822800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.822852 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:53:37.822911 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.822912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.822921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.822964 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.823475 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.823932 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.824602 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.824777 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.824835 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.827389 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.827390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.827390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.827397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.828615 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.828935 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.828937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.828937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.828993 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.829199 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.829200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.829200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.829303 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.829384 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.829600 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.829661 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 14:53:37.830772 22038 tablet.cc:2369] T ec54c2bf385b483babb6ae68f8165595 P 361e21b2071948f68d0110f777e7f242: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:37.851375 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:37.844414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.20.199.1:41235)
0925 14:53:37.844505 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:37.844507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:37.844549 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:53:37.844596 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:37.844597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:37.844605 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:37.844655 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:37.845217 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.845726 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.846477 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:37.846667 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:37.846741 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:37.849087 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:37.849089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:37.849089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:37.849095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:37.850450 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.850737 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.850738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.850738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.850784 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:37.850983 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:37.850983 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:37.850984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:37.851059 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:37.851125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:37.851262 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:53:37.851314 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 14:53:37.851857 21951 tablet_service.cc:2908] Rejecting scan request for tablet ec54c2bf385b483babb6ae68f8165595: Uninitialized: clean time has not yet been initialized
I20260925 14:53:37.852797 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 21908
I20260925 14:53:37.857950 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22042
I20260925 14:53:37.862288 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22176
I20260925 14:53:37.866727 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 21814
2026-09-25T14:53:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1994 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[22346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[22346](info): set up 2 sockets
Sep 25 14:53:37 dist-test-slave-8qt6 krb5kdc[22346](info): commencing operation
krb5kdc: starting...
W20260925 14:53:38.900862 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:38 dist-test-slave-8qt6 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:39.022716 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34231
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:39.103606 22359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:39.103813 22359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:39.103835 22359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:39.104919 22359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:39.104954 22359 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:53:39.104966 22359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:39.104979 22359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:39.104992 22359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:39.105003 22359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:39.106375 22359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34231
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:39.106560 22359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:39.106721 22359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:39.111536 22364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.111542 22365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.111555 22367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:39.111783 22359 server_base.cc:1034] running on GCE node
I20260925 14:53:39.112087 22359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:39.112377 22359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:39.113467 22359 hybrid_clock.cc:648] HybridClock initialized: now 1790348019113434 us; error 46 us; skew 500 ppm
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:39.120095 22359 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:39.120498 22359 webserver.cc:460] Webserver started at http://127.20.199.62:34909/ using document root <none> and password file <none>
I20260925 14:53:39.120731 22359 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:39.120800 22359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:39.120959 22359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:39.122040 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "b627c2e7da544649b13a1aa216ea4b6c"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "c52b1ec35e274d3458d570234ba94107"
server_key_iv: "b8fe7b84a0969bf4901bc392e2e575e4"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.122397 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "b627c2e7da544649b13a1aa216ea4b6c"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "c52b1ec35e274d3458d570234ba94107"
server_key_iv: "b8fe7b84a0969bf4901bc392e2e575e4"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.123662 22359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:39.124435 22374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.124671 22359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:39.124768 22359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "b627c2e7da544649b13a1aa216ea4b6c"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "c52b1ec35e274d3458d570234ba94107"
server_key_iv: "b8fe7b84a0969bf4901bc392e2e575e4"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.124853 22359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:39.136435 22359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:39.138377 22359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:39.138528 22359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:39.142959 22359 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34231
I20260925 14:53:39.143008 22436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34231 every 8 connection(s)
I20260925 14:53:39.143309 22359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:39.143863 22437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.146116 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: Bootstrap starting.
I20260925 14:53:39.146692 22437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.146970 22437 log.cc:824] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:39.147439 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22359
I20260925 14:53:39.147534 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:39.147703 21276 external_mini_cluster.cc:1428] Setting key ef0134e9740d671e72ff5a0961836b2d
I20260925 14:53:39.147732 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: No bootstrap required, opened a new log
I20260925 14:53:39.150188 22437 raft_consensus.cc:348] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b627c2e7da544649b13a1aa216ea4b6c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34231 } }
I20260925 14:53:39.150319 22437 raft_consensus.cc:374] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.150349 22437 raft_consensus.cc:729] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b627c2e7da544649b13a1aa216ea4b6c, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.150486 22437 consensus_queue.cc:260] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b627c2e7da544649b13a1aa216ea4b6c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34231 } }
I20260925 14:53:39.150548 22437 raft_consensus.cc:388] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:39.150569 22437 raft_consensus.cc:482] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:39.150636 22437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.151155 22437 raft_consensus.cc:504] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b627c2e7da544649b13a1aa216ea4b6c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34231 } }
I20260925 14:53:39.151273 22437 leader_election.cc:302] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b627c2e7da544649b13a1aa216ea4b6c; no voters:
I20260925 14:53:39.151504 22437 leader_election.cc:288] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:39.151528 22442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:39.151643 22442 raft_consensus.cc:686] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [term 1 LEADER]: Becoming Leader. State: Replica: b627c2e7da544649b13a1aa216ea4b6c, State: Running, Role: LEADER
I20260925 14:53:39.151762 22437 sys_catalog.cc:564] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:39.151779 22442 consensus_queue.cc:237] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b627c2e7da544649b13a1aa216ea4b6c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34231 } }
I20260925 14:53:39.152119 22442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b627c2e7da544649b13a1aa216ea4b6c. Latest consensus state: current_term: 1 leader_uuid: "b627c2e7da544649b13a1aa216ea4b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b627c2e7da544649b13a1aa216ea4b6c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34231 } } }
I20260925 14:53:39.152129 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b627c2e7da544649b13a1aa216ea4b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b627c2e7da544649b13a1aa216ea4b6c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34231 } } }
I20260925 14:53:39.152259 22442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:39.152293 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:39.152554 22451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:39.153081 22451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:39.155371 22451 catalog_manager.cc:1347] Generated new cluster ID: 2e30463d649c4c8298c793ae50e1ac83
I20260925 14:53:39.155423 22451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:39.159323 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.148235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.20.199.62:34231)
0925 14:53:39.148419 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.148428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.148460 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:53:39.149002 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.149008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.149053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.149219 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.149889 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.150984 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.151959 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.152553 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.152635 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.155496 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.155500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.155500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.155511 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.157493 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.157914 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.157917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.157917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.158016 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.158231 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.158232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.158233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.158518 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.158640 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.158914 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.159058 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260925 14:53:39.163403 22451 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:39.164076 22451 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:39.169595 22451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: Generated new TSK 0
I20260925 14:53:39.169801 22451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:39.200735 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34231
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:39.291508 22463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:39.291697 22463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:39.291720 22463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:39.292874 22463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:39.292917 22463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:39.292936 22463 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:39.292953 22463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:39.294370 22463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:39.294605 22463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:39.294862 22463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:39.300836 22468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.300899 22471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.301115 22469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:39.302479 22463 server_base.cc:1034] running on GCE node
I20260925 14:53:39.302686 22463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:39.302999 22463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:39.304175 22463 hybrid_clock.cc:648] HybridClock initialized: now 1790348019304154 us; error 34 us; skew 500 ppm
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:39.306919 22463 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:39.307331 22463 webserver.cc:460] Webserver started at http://127.20.199.1:43173/ using document root <none> and password file <none>
I20260925 14:53:39.307572 22463 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:39.307641 22463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:39.307789 22463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:39.308841 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "1ac27651a628430dae7fc663bb6a97ce"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "c6b6e6a6c3916931589bef1c6fbe05f4"
server_key_iv: "10122c5323198bc9ff3fd16e0d619c36"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.309203 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "1ac27651a628430dae7fc663bb6a97ce"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "c6b6e6a6c3916931589bef1c6fbe05f4"
server_key_iv: "10122c5323198bc9ff3fd16e0d619c36"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.310549 22463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:39.311280 22478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.311458 22463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:39.311544 22463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "1ac27651a628430dae7fc663bb6a97ce"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "c6b6e6a6c3916931589bef1c6fbe05f4"
server_key_iv: "10122c5323198bc9ff3fd16e0d619c36"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.311630 22463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:39.324093 22463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:39.326048 22463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:39.326203 22463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:39.326710 22463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:39.326772 22463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.326846 22463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:39.326869 22463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.333873 22463 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:35457
I20260925 14:53:39.333951 22593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:35457 every 8 connection(s)
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:39.334265 22463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:39.335930 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22463
I20260925 14:53:39.336040 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:39.336207 21276 external_mini_cluster.cc:1428] Setting key ec9ccc8ce9bb431b72b1c53645942fde
I20260925 14:53:39.337176 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.327244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:52545 (local address 127.20.199.62:34231)
0925 14:53:39.327316 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.327318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.327554 (+ 236us) server_negotiation.cc:407] Connection header received
0925 14:53:39.328126 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.328128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.328139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.328209 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.329345 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.330212 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.331886 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.332110 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.332166 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.334698 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.334700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.334700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.334708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.335814 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.336199 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.336201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.336201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.336257 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.336550 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.336551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.336552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.336672 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.336746 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.337049 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.337116 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:53:39.337492 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.327248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.20.199.1:52545)
0925 14:53:39.327422 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.327483 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.328032 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.328323 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.328354 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.328777 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.329075 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.329081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.330310 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.330311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.331790 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.331793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.332201 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.332203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.332363 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.332964 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.332973 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.334619 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.335893 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.335899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.335902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.336113 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.336345 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.336346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.336346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.336467 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.336747 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.336754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.336903 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.337026 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.337195 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260925 14:53:39.338065 22594 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34231
I20260925 14:53:39.338160 22594 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:39.338356 22594 heartbeater.cc:511] Master 127.20.199.62:34231 requested a full tablet report, sending...
I20260925 14:53:39.338846 22390 ts_manager.cc:194] Registered new tserver with Master: 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1:35457)
I20260925 14:53:39.339680 22390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:52545
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
I20260925 14:53:39.345343 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.339260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:49003 (local address 127.20.199.62:34231)
0925 14:53:39.339303 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.339305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.339413 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:53:39.339468 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.339469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.339478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.339524 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.339914 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.340686 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.341813 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.341986 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.342044 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.342993 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.342994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.342995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.343000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.344280 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.344656 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.344656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.344657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.344701 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.344887 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.344887 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.344888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.345015 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.345065 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.345224 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.345276 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:39.345367 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.339216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.20.199.1:49003)
0925 14:53:39.339344 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.339354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.339410 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.339528 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.339529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.339757 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.339825 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.339826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.340807 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.340808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.341734 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.341737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.342066 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.342068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.342156 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.342530 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.342535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.342911 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.344366 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.344367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.344367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.344581 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.344760 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.344761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.344761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.344826 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.345064 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.345065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.345115 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.345208 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.345305 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:39.364487 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34231
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:39.444190 22603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:39.444389 22603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:39.444412 22603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:39.445609 22603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:39.445650 22603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:39.445670 22603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:39.445688 22603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:39.447064 22603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:39.447285 22603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:39.447525 22603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:39.452453 22603 server_base.cc:1034] running on GCE node
W20260925 14:53:39.452389 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.452410 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.452374 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:39.452914 22603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:39.453186 22603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:39.454344 22603 hybrid_clock.cc:648] HybridClock initialized: now 1790348019454314 us; error 48 us; skew 500 ppm
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:39.457005 22603 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:39.457504 22603 webserver.cc:460] Webserver started at http://127.20.199.2:44421/ using document root <none> and password file <none>
I20260925 14:53:39.457726 22603 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:39.457774 22603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:39.457942 22603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:39.459004 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "8539b8b33bb740e6b91121d514191fc8"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "df005bc43381654cb14dc7e65369a1b4"
server_key_iv: "6133931aadba77c4c117114eed8f16da"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.459384 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "8539b8b33bb740e6b91121d514191fc8"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "df005bc43381654cb14dc7e65369a1b4"
server_key_iv: "6133931aadba77c4c117114eed8f16da"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.460732 22603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:39.461508 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.461699 22603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:39.461792 22603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "8539b8b33bb740e6b91121d514191fc8"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "df005bc43381654cb14dc7e65369a1b4"
server_key_iv: "6133931aadba77c4c117114eed8f16da"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.461871 22603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:39.474011 22603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:39.476361 22603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:39.476526 22603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:39.477027 22603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:39.477073 22603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.477097 22603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:39.477149 22603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.484318 22603 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:40081
I20260925 14:53:39.484378 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:40081 every 8 connection(s)
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:39.484696 22603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:39.487603 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.477513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:41361 (local address 127.20.199.62:34231)
0925 14:53:39.477574 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.477576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.477862 (+ 286us) server_negotiation.cc:407] Connection header received
0925 14:53:39.478331 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.478332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.478344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.478402 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.479254 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.480116 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.481676 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.482020 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.482113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.485203 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.485205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.485205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.485213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.486472 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.486849 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.486850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.486851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.486900 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.487105 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.487105 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.487105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.487203 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.487285 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.487482 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.487540 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:53:39.487768 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.477539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.20.199.2:41361)
0925 14:53:39.477722 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.477778 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.478226 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.478470 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.478473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.478844 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.479127 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.479134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.480220 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.480221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.481563 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.481566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.482130 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.482131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.482273 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.483347 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.483357 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.485097 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.486556 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.486561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.486563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.486777 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.486960 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.486961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.486961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.487051 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.487274 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.487279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.487384 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.487462 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.487573 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260925 14:53:39.488281 22734 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34231
I20260925 14:53:39.488379 22734 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:39.488546 22734 heartbeater.cc:511] Master 127.20.199.62:34231 requested a full tablet report, sending...
I20260925 14:53:39.488915 22391 ts_manager.cc:194] Registered new tserver with Master: 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.489012 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22603
I20260925 14:53:39.489099 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:39.489295 21276 external_mini_cluster.cc:1428] Setting key f52a71ee19ab4f669b67edcc79438b9e
I20260925 14:53:39.489394 22391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:41361
I20260925 14:53:39.495410 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.489368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:45771 (local address 127.20.199.62:34231)
0925 14:53:39.489478 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.489480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.489588 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:53:39.489673 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.489674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.489684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.489711 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.490149 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.490602 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.491541 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.491839 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.491919 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.492825 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.492826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.492826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.492832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.494301 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.494698 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.494699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.494699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.494744 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.494946 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.494947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.494947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.495070 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.495120 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.495299 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.495356 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 14:53:39.495492 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.489310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.20.199.2:45771)
0925 14:53:39.489539 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.489549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.489602 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.489782 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.489783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.489991 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.490062 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.490064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.490699 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.490699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.491465 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.491467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.491913 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.491914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.491967 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.492348 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.492353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.492744 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.494383 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.494384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.494385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.494623 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.494810 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.494810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.494811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.494881 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.495135 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.495136 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.495187 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.495282 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.495383 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:39.514297 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34231
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:39.593799 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:39.593991 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:39.594015 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:39.595073 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:39.595104 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:39.595124 22743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:39.595142 22743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:39.596447 22743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:34231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:39.596611 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:39.596768 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:39.601471 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.601501 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:39.601480 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:39.601740 22743 server_base.cc:1034] running on GCE node
I20260925 14:53:39.601960 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:39.602156 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:39.603289 22743 hybrid_clock.cc:648] HybridClock initialized: now 1790348019603280 us; error 27 us; skew 500 ppm
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:39.605664 22743 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:39.606112 22743 webserver.cc:460] Webserver started at http://127.20.199.3:35479/ using document root <none> and password file <none>
I20260925 14:53:39.606304 22743 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:39.606349 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:39.606437 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:39.607429 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "38ee006bfa794e8595aeb6cb715df6ac"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "9848853cabee57279d5701bca2ffb72f"
server_key_iv: "1f8beaf83fdbab22c5a353318245aaa9"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.607723 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "38ee006bfa794e8595aeb6cb715df6ac"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "9848853cabee57279d5701bca2ffb72f"
server_key_iv: "1f8beaf83fdbab22c5a353318245aaa9"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.608912 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:39.609580 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.609742 22743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.609817 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "38ee006bfa794e8595aeb6cb715df6ac"
format_stamp: "Formatted at 2026-09-25 14:53:39 on dist-test-slave-8qt6"
server_key: "9848853cabee57279d5701bca2ffb72f"
server_key_iv: "1f8beaf83fdbab22c5a353318245aaa9"
server_key_version: "encryptionkey@0"
I20260925 14:53:39.609867 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:39.619830 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:39.621948 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:39.622062 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:39.622483 22743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:39.622519 22743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.622567 22743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:39.622601 22743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.629333 22743 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:34973
I20260925 14:53:39.629436 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:34973 every 8 connection(s)
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:39.629727 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:39.632366 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.623048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:47383 (local address 127.20.199.62:34231)
0925 14:53:39.623120 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.623123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.623265 (+ 142us) server_negotiation.cc:407] Connection header received
0925 14:53:39.623707 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.623709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.623723 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.623808 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.624593 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.625440 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.627063 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.627417 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.627502 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.630203 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.630204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.630204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.630210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.631298 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.631658 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.631659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.631660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.631707 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.631907 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.631908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.631908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.632003 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.632053 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.632250 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.632305 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:53:39.632619 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.622971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.20.199.3:47383)
0925 14:53:39.623140 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.623178 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.623622 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.623878 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.623882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.624199 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.624490 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.624496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.625532 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.625533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.626968 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.626971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.627504 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.627505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.627644 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.628175 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.628185 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.630128 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.631363 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.631367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.631369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.631593 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.631779 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.631779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.631780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.631858 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.632058 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.632062 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.632164 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.632241 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.632350 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 14:53:39.633133 22874 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34231
I20260925 14:53:39.633219 22874 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:39.633450 22874 heartbeater.cc:511] Master 127.20.199.62:34231 requested a full tablet report, sending...
I20260925 14:53:39.633864 22390 ts_manager.cc:194] Registered new tserver with Master: 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973)
I20260925 14:53:39.634370 22390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:47383
I20260925 14:53:39.638657 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22743
I20260925 14:53:39.638787 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:39.638996 21276 external_mini_cluster.cc:1428] Setting key b262af1681c47d0db77d2b9688d59d05
I20260925 14:53:39.639626 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.634250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:44397 (local address 127.20.199.62:34231)
0925 14:53:39.634504 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.634506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.634514 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:39.634532 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.634532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.634543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.634598 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.634964 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.635477 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.636117 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.636287 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.636334 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.637177 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.637178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.637178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.637183 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.638427 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.638715 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.638716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.638716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.638814 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.639075 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.639076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.639077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.639207 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.639271 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.639455 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.639535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260925 14:53:39.639636 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.634183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.20.199.3:44397)
0925 14:53:39.634310 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.634321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.634378 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.634610 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.634611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.634805 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.634872 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.634888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.635558 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.635559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.636052 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.636053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.636351 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.636352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.636403 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.636826 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.636830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.637113 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.638491 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.638491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.638492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.638646 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.638922 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.638923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.638924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.638989 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.639277 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.639278 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.639327 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.639423 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.639526 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260925 14:53:39.639966 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:39.661918 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.655564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.20.199.62:34231)
0925 14:53:39.655621 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.655623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.655638 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:53:39.655659 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.655660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.655668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.655717 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.656232 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.656714 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.657385 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.657607 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.657658 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.659664 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.659665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.659665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.659672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.661004 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.661296 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.661297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.661297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.661337 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.661534 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.661534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.661535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.661626 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.661696 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.661780 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.661832 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:53:39.663379 22390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:39.664857 22390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:39.670758 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.666877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:35457 (local address 127.0.0.1:34882)
0925 14:53:39.667004 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.667047 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.667114 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.667349 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.667352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.667354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.667427 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.667429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.668480 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.668481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.669498 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.669501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.670434 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.670435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.670519 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.670531 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.670622 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.670673 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260925 14:53:39.670770 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.667005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.20.199.1:35457)
0925 14:53:39.667184 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.667189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.667198 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:39.667233 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.667235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.667249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.667347 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.667508 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.668399 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.669591 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.670335 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.670431 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.670492 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.670640 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.670677 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260925 14:53:39.671281 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.666877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34973 (local address 127.0.0.1:44506)
0925 14:53:39.667207 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.667218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.667270 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.667882 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.667883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.667886 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.667924 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.667925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.669123 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.669124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.670096 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.670099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.671051 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.671052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.671116 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.671120 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.671149 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.671186 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":2,"thread_start_us":48,"threads_started":1}
I20260925 14:53:39.671492 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.667253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.20.199.2:40081)
0925 14:53:39.667411 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.667415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.667521 (+ 106us) server_negotiation.cc:407] Connection header received
0925 14:53:39.667581 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.667584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.667599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.667675 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.667882 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.668993 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.670286 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.670972 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.671032 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.671083 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.671325 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.671373 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260925 14:53:39.671622 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.667051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:40081 (local address 127.0.0.1:42308)
0925 14:53:39.667448 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.667457 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.667511 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.667749 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.667750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.667752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.667803 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.667805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.669082 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.669083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.670216 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.670218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.671111 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.671112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.671166 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.671168 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.671224 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.671256 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20260925 14:53:39.671600 22528 tablet_service.cc:1467] Processing CreateTablet for tablet b17accb54ccc4a0f959fc17549700018 (DEFAULT_TABLE table=test-table [id=fd928ae37f494359a1e23a404e2e16e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:39.671803 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.667035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.20.199.3:34973)
0925 14:53:39.667632 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.667636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.667644 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:39.667753 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.667755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.667771 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.667880 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.667993 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.669024 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.670226 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.670944 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.671470 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.671535 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.671562 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.671604 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":82,"threads_started":1}
I20260925 14:53:39.671977 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17accb54ccc4a0f959fc17549700018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.672288 22668 tablet_service.cc:1467] Processing CreateTablet for tablet b17accb54ccc4a0f959fc17549700018 (DEFAULT_TABLE table=test-table [id=fd928ae37f494359a1e23a404e2e16e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:39.672320 22808 tablet_service.cc:1467] Processing CreateTablet for tablet b17accb54ccc4a0f959fc17549700018 (DEFAULT_TABLE table=test-table [id=fd928ae37f494359a1e23a404e2e16e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:39.672607 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17accb54ccc4a0f959fc17549700018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.672607 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17accb54ccc4a0f959fc17549700018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.674922 22894 tablet_bootstrap.cc:492] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Bootstrap starting.
I20260925 14:53:39.675549 22894 tablet_bootstrap.cc:654] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.675719 22895 tablet_bootstrap.cc:492] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Bootstrap starting.
I20260925 14:53:39.676012 22894 log.cc:824] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:39.676388 22896 tablet_bootstrap.cc:492] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Bootstrap starting.
I20260925 14:53:39.676680 22895 tablet_bootstrap.cc:654] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.676724 22894 tablet_bootstrap.cc:492] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: No bootstrap required, opened a new log
I20260925 14:53:39.676836 22894 ts_tablet_manager.cc:1397] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:39.676992 22896 tablet_bootstrap.cc:654] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.677001 22895 log.cc:824] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:39.677306 22896 log.cc:824] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:39.677803 22895 tablet_bootstrap.cc:492] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: No bootstrap required, opened a new log
I20260925 14:53:39.677932 22895 ts_tablet_manager.cc:1397] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:39.678050 22896 tablet_bootstrap.cc:492] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: No bootstrap required, opened a new log
I20260925 14:53:39.678119 22896 ts_tablet_manager.cc:1397] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:39.679046 22894 raft_consensus.cc:348] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.679210 22894 raft_consensus.cc:374] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.679255 22894 raft_consensus.cc:729] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac27651a628430dae7fc663bb6a97ce, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.679371 22894 consensus_queue.cc:260] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.679440 22895 raft_consensus.cc:348] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.679590 22895 raft_consensus.cc:374] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.679617 22894 ts_tablet_manager.cc:1428] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:39.679632 22895 raft_consensus.cc:729] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ee006bfa794e8595aeb6cb715df6ac, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.679736 22594 heartbeater.cc:503] Master 127.20.199.62:34231 was elected leader, sending a full tablet report...
I20260925 14:53:39.679765 22895 consensus_queue.cc:260] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.680027 22895 ts_tablet_manager.cc:1428] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:39.680064 22874 heartbeater.cc:503] Master 127.20.199.62:34231 was elected leader, sending a full tablet report...
I20260925 14:53:39.680114 22896 raft_consensus.cc:348] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.680245 22896 raft_consensus.cc:374] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.680284 22896 raft_consensus.cc:729] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8539b8b33bb740e6b91121d514191fc8, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.680397 22896 consensus_queue.cc:260] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.680661 22896 ts_tablet_manager.cc:1428] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:39.680706 22734 heartbeater.cc:503] Master 127.20.199.62:34231 was elected leader, sending a full tablet report...
I20260925 14:53:39.693642 22900 raft_consensus.cc:482] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:39.693782 22900 raft_consensus.cc:504] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.694097 22900 leader_election.cc:288] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973), 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.697017 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.694321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:54533 (local address 127.20.199.3:34973)
0925 14:53:39.694387 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.694390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.694400 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:39.694428 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.694429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.694440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.694526 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.694670 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.695259 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.696259 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.696651 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.696704 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.696711 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.696926 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.696964 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:53:39.697012 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.694249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34973 (local address 127.20.199.1:54533)
0925 14:53:39.694289 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.694301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.694340 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.694532 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.694533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.694534 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.694597 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.694599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.695333 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.695334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.696168 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.696170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.696731 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.696731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.696855 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.696859 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.696905 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.696943 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":25}
I20260925 14:53:39.697149 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.694475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58577 (local address 127.20.199.2:40081)
0925 14:53:39.694518 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.694520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.694577 (+ 57us) server_negotiation.cc:407] Connection header received
0925 14:53:39.694751 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.694752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.694760 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.694801 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.694969 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.695471 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.696391 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.696767 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.696836 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.696843 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.697040 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.697080 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:53:39.697194 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.694359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:40081 (local address 127.20.199.1:58577)
0925 14:53:39.694518 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.694532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.694599 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.694825 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.694826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.694827 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.694899 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.694901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.695553 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.695554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.696312 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.696314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.696846 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.696847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.696938 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.696942 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.697028 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.697067 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260925 14:53:39.697459 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17accb54ccc4a0f959fc17549700018" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ee006bfa794e8595aeb6cb715df6ac" is_pre_election: true
I20260925 14:53:39.697604 22828 raft_consensus.cc:2454] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 0.
I20260925 14:53:39.697661 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17accb54ccc4a0f959fc17549700018" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8" is_pre_election: true
I20260925 14:53:39.697800 22688 raft_consensus.cc:2454] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 0.
I20260925 14:53:39.697855 22480 leader_election.cc:302] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac27651a628430dae7fc663bb6a97ce, 38ee006bfa794e8595aeb6cb715df6ac; no voters:
I20260925 14:53:39.697986 22900 raft_consensus.cc:2781] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:39.698041 22900 raft_consensus.cc:482] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:39.698061 22900 raft_consensus.cc:3037] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.698675 22900 raft_consensus.cc:504] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.698800 22900 leader_election.cc:288] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 election: Requested vote from peers 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973), 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.698956 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17accb54ccc4a0f959fc17549700018" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ee006bfa794e8595aeb6cb715df6ac"
I20260925 14:53:39.698988 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17accb54ccc4a0f959fc17549700018" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8"
I20260925 14:53:39.699043 22688 raft_consensus.cc:3037] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.699062 22828 raft_consensus.cc:3037] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.699744 22828 raft_consensus.cc:2454] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 1.
I20260925 14:53:39.699744 22688 raft_consensus.cc:2454] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 1.
I20260925 14:53:39.699942 22480 leader_election.cc:302] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac27651a628430dae7fc663bb6a97ce, 38ee006bfa794e8595aeb6cb715df6ac; no voters:
I20260925 14:53:39.700042 22900 raft_consensus.cc:2781] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:39.700197 22900 raft_consensus.cc:686] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 1 LEADER]: Becoming Leader. State: Replica: 1ac27651a628430dae7fc663bb6a97ce, State: Running, Role: LEADER
I20260925 14:53:39.700295 22900 consensus_queue.cc:237] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.701136 22389 catalog_manager.cc:5495] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1). New cstate: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:39.710573 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.707994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.20.199.1:35457)
0925 14:53:39.708066 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.708068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.708104 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:53:39.708185 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.708186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.708200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.708298 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:39.708431 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.708866 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.709647 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.709827 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.709880 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.710066 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:39.710149 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:39.710204 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.710248 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.710264 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:53:39.712186 22828 raft_consensus.cc:1261] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Refusing update from remote peer 1ac27651a628430dae7fc663bb6a97ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:39.712220 22688 raft_consensus.cc:1261] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Refusing update from remote peer 1ac27651a628430dae7fc663bb6a97ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:39.712534 22904 consensus_queue.cc:1035] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:39.712661 22900 consensus_queue.cc:1035] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:39.716580 22909 mvcc.cc:205] Tried to move back new op lower bound from 7333265488738656256 to 7333265488693018624. Current Snapshot: MvccSnapshot[applied={T|T < 7333265488738656256}]
I20260925 14:53:39.718223 22390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:39.718312 22390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:39.719050 22390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: Sending DeleteTablet for 3 replicas of tablet b17accb54ccc4a0f959fc17549700018
I20260925 14:53:39.719362 22528 tablet_service.cc:1514] Processing DeleteTablet for tablet b17accb54ccc4a0f959fc17549700018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:39 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:34882
I20260925 14:53:39.719471 22668 tablet_service.cc:1514] Processing DeleteTablet for tablet b17accb54ccc4a0f959fc17549700018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:39 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:42308
I20260925 14:53:39.719471 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet b17accb54ccc4a0f959fc17549700018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:39 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44506
I20260925 14:53:39.719734 22390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:39.719846 22390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:39.721030 22528 tablet_service.cc:1467] Processing CreateTablet for tablet 74eaa7467fa044d09c61fd78148d177c (DEFAULT_TABLE table=test-table [id=6dfd7f3e164b456eb47bfaacba5929b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:39.721038 22668 tablet_service.cc:1467] Processing CreateTablet for tablet 74eaa7467fa044d09c61fd78148d177c (DEFAULT_TABLE table=test-table [id=6dfd7f3e164b456eb47bfaacba5929b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:39.721127 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eaa7467fa044d09c61fd78148d177c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.721154 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eaa7467fa044d09c61fd78148d177c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.722236 22894 tablet_bootstrap.cc:492] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Bootstrap starting.
I20260925 14:53:39.722450 22896 tablet_bootstrap.cc:492] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Bootstrap starting.
I20260925 14:53:39.722645 22894 tablet_bootstrap.cc:654] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.723178 22896 tablet_bootstrap.cc:654] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.723167 22808 tablet_service.cc:1467] Processing CreateTablet for tablet 74eaa7467fa044d09c61fd78148d177c (DEFAULT_TABLE table=test-table [id=6dfd7f3e164b456eb47bfaacba5929b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:39.723281 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eaa7467fa044d09c61fd78148d177c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.723443 22894 tablet_bootstrap.cc:492] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: No bootstrap required, opened a new log
I20260925 14:53:39.723482 22894 ts_tablet_manager.cc:1397] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:39.723620 22894 raft_consensus.cc:348] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.723675 22894 raft_consensus.cc:374] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.723694 22894 raft_consensus.cc:729] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac27651a628430dae7fc663bb6a97ce, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.723734 22894 consensus_queue.cc:260] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.723820 22894 ts_tablet_manager.cc:1428] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.723850 22914 raft_consensus.cc:482] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:39.723901 22914 raft_consensus.cc:504] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.724193 22914 leader_election.cc:288] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081), 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973)
I20260925 14:53:39.724557 22895 tablet_bootstrap.cc:492] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Bootstrap starting.
W20260925 14:53:39.724565 22480 leader_election.cc:341] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973): Not found: Tablet not found: 74eaa7467fa044d09c61fd78148d177c
I20260925 14:53:39.724625 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eaa7467fa044d09c61fd78148d177c" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8" is_pre_election: true
W20260925 14:53:39.724854 22480 leader_election.cc:341] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081): Illegal state: must be running to vote when last-logged opid is not known
I20260925 14:53:39.724918 22480 leader_election.cc:302] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ac27651a628430dae7fc663bb6a97ce; no voters: 38ee006bfa794e8595aeb6cb715df6ac, 8539b8b33bb740e6b91121d514191fc8
I20260925 14:53:39.725075 22895 tablet_bootstrap.cc:654] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.725610 22896 tablet_bootstrap.cc:492] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: No bootstrap required, opened a new log
I20260925 14:53:39.725669 22896 ts_tablet_manager.cc:1397] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:53:39.725732 22895 tablet_bootstrap.cc:492] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: No bootstrap required, opened a new log
I20260925 14:53:39.725780 22895 ts_tablet_manager.cc:1397] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:39.725811 22896 raft_consensus.cc:348] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.725879 22896 raft_consensus.cc:374] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.725906 22896 raft_consensus.cc:729] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8539b8b33bb740e6b91121d514191fc8, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.725932 22895 raft_consensus.cc:348] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.725955 22896 consensus_queue.cc:260] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.725999 22895 raft_consensus.cc:374] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.726027 22895 raft_consensus.cc:729] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ee006bfa794e8595aeb6cb715df6ac, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.726034 22917 tablet_replica.cc:330] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: stopping tablet replica
I20260925 14:53:39.726056 22896 ts_tablet_manager.cc:1428] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.726066 22895 consensus_queue.cc:260] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.726127 22917 raft_consensus.cc:2229] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:39.726157 22895 ts_tablet_manager.cc:1428] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:39.726225 22917 raft_consensus.cc:2258] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:39.726233 22914 raft_consensus.cc:2726] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 14:53:39.726679 22917 ts_tablet_manager.cc:1905] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:39.728147 22917 ts_tablet_manager.cc:1918] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:39.728216 22917 log.cc:1196] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/b17accb54ccc4a0f959fc17549700018
I20260925 14:53:39.728508 22917 ts_tablet_manager.cc:1939] T b17accb54ccc4a0f959fc17549700018 P 1ac27651a628430dae7fc663bb6a97ce: Deleting consensus metadata
I20260925 14:53:39.728905 22375 catalog_manager.cc:4841] TS 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1:35457): tablet b17accb54ccc4a0f959fc17549700018 (table test-table [id=fd928ae37f494359a1e23a404e2e16e1]) successfully deleted
I20260925 14:53:39.729533 22918 tablet_replica.cc:330] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: stopping tablet replica
I20260925 14:53:39.729658 22918 raft_consensus.cc:2229] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:39.729785 22918 raft_consensus.cc:2258] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:39.729985 22919 tablet_replica.cc:330] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: stopping tablet replica
I20260925 14:53:39.730084 22919 raft_consensus.cc:2229] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:39.730199 22919 raft_consensus.cc:2258] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:39.730630 22919 ts_tablet_manager.cc:1905] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:39.732060 22919 ts_tablet_manager.cc:1918] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:39.732161 22919 log.cc:1196] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/b17accb54ccc4a0f959fc17549700018
I20260925 14:53:39.732409 22919 ts_tablet_manager.cc:1939] T b17accb54ccc4a0f959fc17549700018 P 8539b8b33bb740e6b91121d514191fc8: Deleting consensus metadata
I20260925 14:53:39.732975 22918 ts_tablet_manager.cc:1905] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:39.733016 22376 catalog_manager.cc:4841] TS 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081): tablet b17accb54ccc4a0f959fc17549700018 (table test-table [id=fd928ae37f494359a1e23a404e2e16e1]) successfully deleted
I20260925 14:53:39.734284 22918 ts_tablet_manager.cc:1918] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:39.734366 22918 log.cc:1196] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/b17accb54ccc4a0f959fc17549700018
I20260925 14:53:39.734612 22918 ts_tablet_manager.cc:1939] T b17accb54ccc4a0f959fc17549700018 P 38ee006bfa794e8595aeb6cb715df6ac: Deleting consensus metadata
I20260925 14:53:39.734995 22376 catalog_manager.cc:4841] TS 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973): tablet b17accb54ccc4a0f959fc17549700018 (table test-table [id=fd928ae37f494359a1e23a404e2e16e1]) successfully deleted
W20260925 14:53:39.735296 22735 tablet.cc:2369] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:39.742650 22901 raft_consensus.cc:482] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:39.742717 22901 raft_consensus.cc:504] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.742988 22901 leader_election.cc:288] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1:35457), 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.746925 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.743254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:37579 (local address 127.20.199.2:40081)
0925 14:53:39.743296 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.743298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.743548 (+ 250us) server_negotiation.cc:407] Connection header received
0925 14:53:39.743604 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.743605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.743614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.743661 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.743808 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.744486 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.745929 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.746498 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.746580 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.746589 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.746819 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.746860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:39.746963 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.743165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:40081 (local address 127.20.199.3:37579)
0925 14:53:39.743487 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.743495 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.743556 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.743690 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.743690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.743692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.743746 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.743747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.744562 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.744562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.745843 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.745845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.746604 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.746605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.746672 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.746677 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.746808 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.746853 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":160,"thread_start_us":54,"threads_started":1}
I20260925 14:53:39.747375 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eaa7467fa044d09c61fd78148d177c" candidate_uuid: "38ee006bfa794e8595aeb6cb715df6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8" is_pre_election: true
I20260925 14:53:39.747464 22688 raft_consensus.cc:2454] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ee006bfa794e8595aeb6cb715df6ac in term 0.
I20260925 14:53:39.747538 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.743165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:35457 (local address 127.20.199.3:46971)
0925 14:53:39.743202 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.743213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.743249 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.743520 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.743521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.743523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:39.743586 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.743588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.744519 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.744520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.745842 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.745845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.747275 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.747276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.747363 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.747374 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.747428 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.747462 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":57}
I20260925 14:53:39.747565 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.743276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:46971 (local address 127.20.199.1:35457)
0925 14:53:39.743340 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.743342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.743350 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:39.743372 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.743373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.743382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.743436 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:39.743656 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.744431 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.746622 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.747169 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.747236 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.747244 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.747438 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.747485 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:53:39.747625 22760 leader_election.cc:302] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ee006bfa794e8595aeb6cb715df6ac, 8539b8b33bb740e6b91121d514191fc8; no voters:
I20260925 14:53:39.747761 22901 raft_consensus.cc:2781] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:39.747797 22901 raft_consensus.cc:482] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:39.747824 22901 raft_consensus.cc:3037] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.747949 22548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eaa7467fa044d09c61fd78148d177c" candidate_uuid: "38ee006bfa794e8595aeb6cb715df6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac27651a628430dae7fc663bb6a97ce" is_pre_election: true
I20260925 14:53:39.748073 22548 raft_consensus.cc:2454] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ee006bfa794e8595aeb6cb715df6ac in term 0.
I20260925 14:53:39.748514 22901 raft_consensus.cc:504] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.748662 22901 leader_election.cc:288] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [CANDIDATE]: Term 1 election: Requested vote from peers 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1:35457), 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.748816 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eaa7467fa044d09c61fd78148d177c" candidate_uuid: "38ee006bfa794e8595aeb6cb715df6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8"
I20260925 14:53:39.748811 22548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eaa7467fa044d09c61fd78148d177c" candidate_uuid: "38ee006bfa794e8595aeb6cb715df6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac27651a628430dae7fc663bb6a97ce"
I20260925 14:53:39.748883 22688 raft_consensus.cc:3037] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.748890 22548 raft_consensus.cc:3037] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.749614 22688 raft_consensus.cc:2454] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ee006bfa794e8595aeb6cb715df6ac in term 1.
I20260925 14:53:39.749750 22548 raft_consensus.cc:2454] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ee006bfa794e8595aeb6cb715df6ac in term 1.
I20260925 14:53:39.749807 22760 leader_election.cc:302] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ee006bfa794e8595aeb6cb715df6ac, 8539b8b33bb740e6b91121d514191fc8; no voters:
I20260925 14:53:39.749898 22901 raft_consensus.cc:2781] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:39.750042 22901 raft_consensus.cc:686] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 1 LEADER]: Becoming Leader. State: Replica: 38ee006bfa794e8595aeb6cb715df6ac, State: Running, Role: LEADER
I20260925 14:53:39.750111 22901 consensus_queue.cc:237] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } }
I20260925 14:53:39.750829 22389 catalog_manager.cc:5495] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3). New cstate: current_term: 1 leader_uuid: "38ee006bfa794e8595aeb6cb715df6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } health_report { overall_health: HEALTHY } } }
Sep 25 14:53:39 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:39.793357 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.786180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.20.199.62:34231)
0925 14:53:39.786234 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:39.786236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:39.786354 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:53:39.786430 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:39.786431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:39.786441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:39.786529 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:39.786983 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.787497 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.788180 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.788428 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.788508 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.790783 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:39.790784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:39.790785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:39.790794 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:39.792140 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.792529 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.792530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.792530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.792584 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:39.792783 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:39.792784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:39.792785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:39.792891 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:39.792967 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:39.793114 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:53:39.793221 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:53:39.793547 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:39.786126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34231 (local address 127.0.0.1:60124)
0925 14:53:39.786288 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:39.786297 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:39.786373 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:39.786533 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:39.786534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:39.786841 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:39.786888 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.786890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.787568 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.787569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.788102 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:39.788103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:39.788509 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:39.788510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:39.788567 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:39.789099 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:39.789103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:39.790678 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:39.792239 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.792246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.792249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.792443 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.792648 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:39.792648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:39.792649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:39.792704 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:39.792957 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:39.792963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:39.793006 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:53:39.793092 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:53:39.793277 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":89,"thread_start_us":72,"threads_started":1}
I20260925 14:53:39.795782 22389 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:60124:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:53:39.795979 22389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:39.797328 22808 tablet_service.cc:1467] Processing CreateTablet for tablet 22424c4295ca40119ca005423ad41aed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68d9dc1af8e248cd9732e0ff98787c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:39.797370 22528 tablet_service.cc:1467] Processing CreateTablet for tablet 22424c4295ca40119ca005423ad41aed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68d9dc1af8e248cd9732e0ff98787c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:39.797376 22668 tablet_service.cc:1467] Processing CreateTablet for tablet 22424c4295ca40119ca005423ad41aed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68d9dc1af8e248cd9732e0ff98787c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:39.797536 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22424c4295ca40119ca005423ad41aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.797536 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22424c4295ca40119ca005423ad41aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.797538 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22424c4295ca40119ca005423ad41aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:39.798959 22894 tablet_bootstrap.cc:492] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: Bootstrap starting.
I20260925 14:53:39.798959 22896 tablet_bootstrap.cc:492] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: Bootstrap starting.
I20260925 14:53:39.799441 22896 tablet_bootstrap.cc:654] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.799626 22894 tablet_bootstrap.cc:654] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.799973 22895 tablet_bootstrap.cc:492] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: Bootstrap starting.
I20260925 14:53:39.800386 22895 tablet_bootstrap.cc:654] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:39.800406 22894 tablet_bootstrap.cc:492] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: No bootstrap required, opened a new log
I20260925 14:53:39.800443 22894 ts_tablet_manager.cc:1397] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:39.800598 22894 raft_consensus.cc:348] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.800701 22894 raft_consensus.cc:374] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.800727 22894 raft_consensus.cc:729] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac27651a628430dae7fc663bb6a97ce, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.800792 22894 consensus_queue.cc:260] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.800904 22894 ts_tablet_manager.cc:1428] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:39.800926 22914 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:39.801016 22914 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:39.801169 22896 tablet_bootstrap.cc:492] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: No bootstrap required, opened a new log
I20260925 14:53:39.801200 22896 ts_tablet_manager.cc:1397] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:39.801314 22896 raft_consensus.cc:348] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.801363 22896 raft_consensus.cc:374] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.801376 22896 raft_consensus.cc:729] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8539b8b33bb740e6b91121d514191fc8, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.801472 22896 consensus_queue.cc:260] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.801592 22896 ts_tablet_manager.cc:1428] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.801640 22902 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:39.801740 22902 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:39.802400 22895 tablet_bootstrap.cc:492] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: No bootstrap required, opened a new log
I20260925 14:53:39.802441 22895 ts_tablet_manager.cc:1397] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:53:39.802595 22895 raft_consensus.cc:348] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.802659 22895 raft_consensus.cc:374] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:39.802672 22895 raft_consensus.cc:729] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ee006bfa794e8595aeb6cb715df6ac, State: Initialized, Role: FOLLOWER
I20260925 14:53:39.802728 22895 consensus_queue.cc:260] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.802822 22895 ts_tablet_manager.cc:1428] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:39.802867 22901 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:39.802922 22901 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:53:39.834898 22595 tablet.cc:2369] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:53:39.880273 22875 tablet.cc:2369] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:39.978417 22914 raft_consensus.cc:482] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:39.978551 22914 raft_consensus.cc:504] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.978781 22914 leader_election.cc:288] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973), 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.979029 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22424c4295ca40119ca005423ad41aed" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8" is_pre_election: true
I20260925 14:53:39.979161 22688 raft_consensus.cc:2454] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 0.
I20260925 14:53:39.979318 22480 leader_election.cc:302] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac27651a628430dae7fc663bb6a97ce, 8539b8b33bb740e6b91121d514191fc8; no voters:
I20260925 14:53:39.979432 22914 raft_consensus.cc:2781] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:39.979406 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22424c4295ca40119ca005423ad41aed" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ee006bfa794e8595aeb6cb715df6ac" is_pre_election: true
I20260925 14:53:39.979471 22914 raft_consensus.cc:482] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:39.979502 22828 raft_consensus.cc:2454] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 0.
I20260925 14:53:39.979523 22914 raft_consensus.cc:3037] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.980242 22914 raft_consensus.cc:504] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.980377 22914 leader_election.cc:288] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 election: Requested vote from peers 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973), 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081)
I20260925 14:53:39.980544 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22424c4295ca40119ca005423ad41aed" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ee006bfa794e8595aeb6cb715df6ac"
I20260925 14:53:39.980562 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22424c4295ca40119ca005423ad41aed" candidate_uuid: "1ac27651a628430dae7fc663bb6a97ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8539b8b33bb740e6b91121d514191fc8"
I20260925 14:53:39.980625 22688 raft_consensus.cc:3037] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.980625 22828 raft_consensus.cc:3037] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:39.981350 22688 raft_consensus.cc:2454] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 1.
I20260925 14:53:39.981431 22828 raft_consensus.cc:2454] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac27651a628430dae7fc663bb6a97ce in term 1.
I20260925 14:53:39.981568 22480 leader_election.cc:302] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac27651a628430dae7fc663bb6a97ce, 38ee006bfa794e8595aeb6cb715df6ac; no voters:
I20260925 14:53:39.981688 22914 raft_consensus.cc:2781] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:39.981745 22914 raft_consensus.cc:686] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [term 1 LEADER]: Becoming Leader. State: Replica: 1ac27651a628430dae7fc663bb6a97ce, State: Running, Role: LEADER
I20260925 14:53:39.981792 22914 consensus_queue.cc:237] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } }
I20260925 14:53:39.982074 22904 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: TxnStatusTablet state changed. Reason: New leader 1ac27651a628430dae7fc663bb6a97ce. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:39.982163 22904 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:39.982311 22936 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:53:39.982426 22936 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:53:39.982445 22389 catalog_manager.cc:5495] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1). New cstate: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:40.029201 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.026379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:35457 (local address 127.0.0.1:34914)
0925 14:53:40.026504 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:40.026516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:40.026571 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:40.026750 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:40.026751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:40.026752 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:40.026806 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:40.026809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.027469 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.027470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:40.028239 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:40.028242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.028719 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.028720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:40.028792 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.028808 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.029012 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.029018 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:53:40.029071 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:53:40.029116 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 14:53:40.029181 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.026452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.20.199.1:35457)
0925 14:53:40.026516 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:40.026519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:40.026550 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:53:40.026651 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:40.026653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:40.026663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:40.026737 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:40.026862 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.027391 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.028341 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.028588 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.028715 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.028907 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.028947 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.029006 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:40.029073 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:53:40.029098 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:40.030354 22688 raft_consensus.cc:1261] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Refusing update from remote peer 1ac27651a628430dae7fc663bb6a97ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:40.030459 22828 raft_consensus.cc:1261] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Refusing update from remote peer 1ac27651a628430dae7fc663bb6a97ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:40.030587 22914 consensus_queue.cc:1035] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:40.030704 22904 consensus_queue.cc:1035] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:40.031046 22902 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: TxnStatusTablet state changed. Reason: New leader 1ac27651a628430dae7fc663bb6a97ce. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:40.031147 22902 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:40.031208 22901 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: TxnStatusTablet state changed. Reason: New leader 1ac27651a628430dae7fc663bb6a97ce. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:40.031287 22901 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:40.032063 22914 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:40.032140 22914 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:40.032188 22907 mvcc.cc:205] Tried to move back new op lower bound from 7333265490042675200 to 7333265489845891072. Current Snapshot: MvccSnapshot[applied={T|T < 7333265490042675200 or (T in {7333265490042675200})}]
I20260925 14:53:40.032629 22904 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:40.032723 22904 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 1ac27651a628430dae7fc663bb6a97ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:40.032945 22901 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:40.033025 22901 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 38ee006bfa794e8595aeb6cb715df6ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:40.033144 22908 mvcc.cc:205] Tried to move back new op lower bound from 7333265490042675200 to 7333265489845891072. Current Snapshot: MvccSnapshot[applied={T|T < 7333265490042675200}]
I20260925 14:53:40.033257 22902 tablet_replica.cc:439] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ac27651a628430dae7fc663bb6a97ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ee006bfa794e8595aeb6cb715df6ac" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34973 } } peers { permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 } } peers { permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 } } }
I20260925 14:53:40.033330 22902 tablet_replica.cc:442] T 22424c4295ca40119ca005423ad41aed P 8539b8b33bb740e6b91121d514191fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:40.036770 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.034287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.20.199.3:34973)
0925 14:53:40.034341 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:40.034343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:40.034350 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:53:40.034386 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:40.034386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:40.034401 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:40.034453 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:40.034599 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.035034 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.036034 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.036220 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.036271 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.036445 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.036535 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.036586 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:40.036673 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 14:53:40.036688 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:53:40.041369 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.039103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:35457 (local address 127.20.199.3:35485)
0925 14:53:40.039202 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:40.039213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:40.039262 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:40.039386 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:40.039387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:40.039390 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:40.039443 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:40.039444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.040047 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.040048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:40.040598 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:40.040599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.040947 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.040948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:40.041015 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.041028 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.041223 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.041225 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:40.041266 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 14:53:40.041307 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 14:53:40.041364 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.039204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:35485 (local address 127.20.199.1:35457)
0925 14:53:40.039246 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:40.039248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:40.039256 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:40.039315 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:40.039317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:40.039327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:40.039381 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:40.039510 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.039967 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.040690 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.040875 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.040949 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.041115 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.041152 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.041217 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:40.041278 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:53:40.041296 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:40.043551 22688 raft_consensus.cc:1261] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Refusing update from remote peer 38ee006bfa794e8595aeb6cb715df6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:40.043623 22548 raft_consensus.cc:1261] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Refusing update from remote peer 38ee006bfa794e8595aeb6cb715df6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:40.043793 22901 consensus_queue.cc:1035] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8539b8b33bb740e6b91121d514191fc8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:40.043918 22924 consensus_queue.cc:1035] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac27651a628430dae7fc663bb6a97ce" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:40.052181 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.049588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:55431 (local address 127.20.199.3:34973)
0925 14:53:40.049628 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:40.049629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:40.049692 (+ 63us) server_negotiation.cc:407] Connection header received
0925 14:53:40.049746 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:40.049747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:40.049762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:40.049803 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:40.049927 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.050358 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.051449 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.051719 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.051766 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.051993 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.052019 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.052059 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:40.052117 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:53:40.052132 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:40.052285 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.049448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34973 (local address 127.20.199.1:55431)
0925 14:53:40.049619 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:40.049631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:40.049696 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:40.049808 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:40.049809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:40.049810 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:40.049865 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:40.049867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.050435 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.050437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:40.051328 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:40.051330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.051827 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.051827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:40.051896 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.051911 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:40.052063 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:40.052065 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:40.052115 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:53:40.052160 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260925 14:53:40.061457 22389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:40.061571 22389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:40.062125 22389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b627c2e7da544649b13a1aa216ea4b6c: Sending DeleteTablet for 3 replicas of tablet 74eaa7467fa044d09c61fd78148d177c
I20260925 14:53:40.062390 22668 tablet_service.cc:1514] Processing DeleteTablet for tablet 74eaa7467fa044d09c61fd78148d177c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:40 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:42308
I20260925 14:53:40.062404 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet 74eaa7467fa044d09c61fd78148d177c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:40 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44506
I20260925 14:53:40.062495 22919 tablet_replica.cc:330] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: stopping tablet replica
I20260925 14:53:40.062500 22918 tablet_replica.cc:330] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: stopping tablet replica
I20260925 14:53:40.062544 22918 raft_consensus.cc:2229] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:40.062547 22919 raft_consensus.cc:2229] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:40.062580 22919 raft_consensus.cc:2258] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:40.062618 22918 raft_consensus.cc:2258] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:40.062894 22919 ts_tablet_manager.cc:1905] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:40.062922 22918 ts_tablet_manager.cc:1905] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:40.062974 22528 tablet_service.cc:1514] Processing DeleteTablet for tablet 74eaa7467fa044d09c61fd78148d177c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:40 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:34882
I20260925 14:53:40.064224 22918 ts_tablet_manager.cc:1918] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:40.064304 22918 log.cc:1196] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/74eaa7467fa044d09c61fd78148d177c
I20260925 14:53:40.064551 22918 ts_tablet_manager.cc:1939] T 74eaa7467fa044d09c61fd78148d177c P 38ee006bfa794e8595aeb6cb715df6ac: Deleting consensus metadata
I20260925 14:53:40.064564 22919 ts_tablet_manager.cc:1918] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:40.064616 22919 log.cc:1196] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/74eaa7467fa044d09c61fd78148d177c
I20260925 14:53:40.064847 22919 ts_tablet_manager.cc:1939] T 74eaa7467fa044d09c61fd78148d177c P 8539b8b33bb740e6b91121d514191fc8: Deleting consensus metadata
I20260925 14:53:40.064849 22376 catalog_manager.cc:4841] TS 38ee006bfa794e8595aeb6cb715df6ac (127.20.199.3:34973): tablet 74eaa7467fa044d09c61fd78148d177c (table test-table [id=6dfd7f3e164b456eb47bfaacba5929b1]) successfully deleted
I20260925 14:53:40.065009 22917 tablet_replica.cc:330] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: stopping tablet replica
I20260925 14:53:40.065080 22917 raft_consensus.cc:2229] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:40.065121 22917 raft_consensus.cc:2258] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:40.065332 22376 catalog_manager.cc:4841] TS 8539b8b33bb740e6b91121d514191fc8 (127.20.199.2:40081): tablet 74eaa7467fa044d09c61fd78148d177c (table test-table [id=6dfd7f3e164b456eb47bfaacba5929b1]) successfully deleted
I20260925 14:53:40.065367 22917 ts_tablet_manager.cc:1905] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:40.066627 22917 ts_tablet_manager.cc:1918] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:40.066707 22917 log.cc:1196] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/74eaa7467fa044d09c61fd78148d177c
I20260925 14:53:40.066897 22917 ts_tablet_manager.cc:1939] T 74eaa7467fa044d09c61fd78148d177c P 1ac27651a628430dae7fc663bb6a97ce: Deleting consensus metadata
I20260925 14:53:40.067185 22375 catalog_manager.cc:4841] TS 1ac27651a628430dae7fc663bb6a97ce (127.20.199.1:35457): tablet 74eaa7467fa044d09c61fd78148d177c (table test-table [id=6dfd7f3e164b456eb47bfaacba5929b1]) successfully deleted
Sep 25 14:53:40 dist-test-slave-8qt6 krb5kdc[22346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:40.077766 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.069669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.20.199.1:35457)
0925 14:53:40.069711 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:40.069713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:40.069887 (+ 174us) server_negotiation.cc:407] Connection header received
0925 14:53:40.069934 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:40.069935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:40.069945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:40.070000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:40.070553 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.071020 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.071759 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.071960 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.072042 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.074959 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:40.074967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:40.074968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:40.074975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:40.076545 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:40.076920 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:40.076923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:40.076924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:40.076983 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:40.077194 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:40.077195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:40.077195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:40.077328 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:40.077500 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:40.077585 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 14:53:40.077662 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:53:40.078059 22588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34930
I20260925 14:53:40.084861 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:40.078959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.20.199.62:34231)
0925 14:53:40.079031 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:40.079033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:40.079083 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:53:40.079156 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:40.079156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:40.079166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:40.079221 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:40.079750 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.080222 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.080899 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:40.081105 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:40.081188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:40.082331 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:40.082333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:40.082333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:40.082340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:40.083758 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:40.084095 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:40.084096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:40.084097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:40.084154 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:40.084350 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:40.084351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:40.084352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:40.084459 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:40.084558 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:40.084665 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:53:40.084743 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 14:53:40.085053 22389 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60138
I20260925 14:53:40.085700 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22463
I20260925 14:53:40.092820 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22603
I20260925 14:53:40.098428 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22743
I20260925 14:53:40.103091 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22359
2026-09-25T14:53:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2236 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:40 dist-test-slave-8qt6 krb5kdc[22956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:40 dist-test-slave-8qt6 krb5kdc[22956](info): set up 2 sockets
Sep 25 14:53:40 dist-test-slave-8qt6 krb5kdc[22956](info): commencing operation
krb5kdc: starting...
W20260925 14:53:41.129037 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:41.246335 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46695
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:46695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:41.326215 22969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:41.326442 22969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:41.326478 22969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:41.327622 22969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:41.327692 22969 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:53:41.327716 22969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:41.327749 22969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:41.327773 22969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:41.327802 22969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:41.329250 22969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:46695
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:41.329541 22969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:41.329766 22969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:41.334424 22977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.334452 22975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.334424 22974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:41.334823 22969 server_base.cc:1034] running on GCE node
I20260925 14:53:41.335071 22969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:41.335374 22969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:41.336526 22969 hybrid_clock.cc:648] HybridClock initialized: now 1790348021336518 us; error 37 us; skew 500 ppm
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:41.338965 22969 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:41.339402 22969 webserver.cc:460] Webserver started at http://127.20.199.62:33477/ using document root <none> and password file <none>
I20260925 14:53:41.339581 22969 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:41.339619 22969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:41.339705 22969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:41.340658 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "23675b867b914828b042926bd23db129"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "803d59cf7346b40e72a4fd5619e4afad"
server_key_iv: "f27c9b015307c0e2b26ec387c61476d4"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.340950 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "23675b867b914828b042926bd23db129"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "803d59cf7346b40e72a4fd5619e4afad"
server_key_iv: "f27c9b015307c0e2b26ec387c61476d4"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.342281 22969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:41.342990 22984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.343209 22969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:41.343291 22969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "23675b867b914828b042926bd23db129"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "803d59cf7346b40e72a4fd5619e4afad"
server_key_iv: "f27c9b015307c0e2b26ec387c61476d4"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.343343 22969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:41.356204 22969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:41.358667 22969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:41.358788 22969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:41.363353 22969 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:46695
I20260925 14:53:41.363449 23046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:46695 every 8 connection(s)
I20260925 14:53:41.363758 22969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:41.364454 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:41.366741 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Bootstrap starting.
I20260925 14:53:41.367308 23047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:41.367594 23047 log.cc:824] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:41.368265 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: No bootstrap required, opened a new log
I20260925 14:53:41.370400 23047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23675b867b914828b042926bd23db129" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46695 } }
I20260925 14:53:41.370559 23047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:41.370613 23047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23675b867b914828b042926bd23db129, State: Initialized, Role: FOLLOWER
I20260925 14:53:41.370808 23047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23675b867b914828b042926bd23db129" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46695 } }
I20260925 14:53:41.370855 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 22969
I20260925 14:53:41.370911 23047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:41.370922 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:41.370952 23047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:41.371001 23047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:41.371043 21276 external_mini_cluster.cc:1428] Setting key aa1773e5596c9e24588ed77c33ce8587
I20260925 14:53:41.371574 23047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23675b867b914828b042926bd23db129" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46695 } }
I20260925 14:53:41.371690 23047 leader_election.cc:302] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23675b867b914828b042926bd23db129; no voters:
I20260925 14:53:41.371833 23047 leader_election.cc:288] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:41.371863 23052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:41.371984 23052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [term 1 LEADER]: Becoming Leader. State: Replica: 23675b867b914828b042926bd23db129, State: Running, Role: LEADER
I20260925 14:53:41.372100 23047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:41.372073 23052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23675b867b914828b042926bd23db129" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46695 } }
I20260925 14:53:41.372412 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23675b867b914828b042926bd23db129. Latest consensus state: current_term: 1 leader_uuid: "23675b867b914828b042926bd23db129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23675b867b914828b042926bd23db129" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46695 } } }
I20260925 14:53:41.372488 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:41.372648 23053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23675b867b914828b042926bd23db129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23675b867b914828b042926bd23db129" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46695 } } }
I20260925 14:53:41.372709 23053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:41.372831 23058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:41.373557 23058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:41.375136 23058 catalog_manager.cc:1347] Generated new cluster ID: a0c7c567e36f4452930f030394d1df33
I20260925 14:53:41.375188 23058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:41.382865 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.371532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.20.199.62:46695)
0925 14:53:41.371727 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.371737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.371772 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:41.371891 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.371899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.371961 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.372934 (+ 973us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.373045 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.374522 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.375614 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.376241 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.376342 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.379175 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.379179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.379179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.379191 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.381207 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.381606 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.381609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.381609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.381738 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.381946 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.381947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.381948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.382261 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.382431 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.382502 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.382651 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260925 14:53:41.384861 23058 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:41.385299 23058 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:41.388963 23058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Generated new TSK 0
I20260925 14:53:41.389132 23058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:41.418529 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46695
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:41.499373 23073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:41.499565 23073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:41.499589 23073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:41.500723 23073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:41.500767 23073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:41.500787 23073 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:53:41.500799 23073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:41.500823 23073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:41.502259 23073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:46695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:41.502496 23073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:41.502743 23073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:41.507566 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.507577 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:41.507761 23073 server_base.cc:1034] running on GCE node
W20260925 14:53:41.507778 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:41.507998 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:41.508234 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:41.509440 23073 hybrid_clock.cc:648] HybridClock initialized: now 1790348021509431 us; error 35 us; skew 500 ppm
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:41.511842 23073 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:41.512249 23073 webserver.cc:460] Webserver started at http://127.20.199.1:45359/ using document root <none> and password file <none>
I20260925 14:53:41.512465 23073 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:41.512532 23073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:41.512693 23073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:41.513798 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "3bfd072c467641189d2fd36b9dff47cc"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "5cf8a8875bbb8d118aa64da4273665d4"
server_key_iv: "1fc276ad27ba8a89c704e810191fba10"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.514153 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "3bfd072c467641189d2fd36b9dff47cc"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "5cf8a8875bbb8d118aa64da4273665d4"
server_key_iv: "1fc276ad27ba8a89c704e810191fba10"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.515430 23073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:41.516135 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.516295 23073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:41.516388 23073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "3bfd072c467641189d2fd36b9dff47cc"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "5cf8a8875bbb8d118aa64da4273665d4"
server_key_iv: "1fc276ad27ba8a89c704e810191fba10"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.516467 23073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:41.534721 23073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:41.536428 23073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:41.536587 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:41.537091 23073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:41.537154 23073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.537191 23073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:41.537220 23073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:41.544273 23073 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:37845
I20260925 14:53:41.544389 23203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:37845 every 8 connection(s)
I20260925 14:53:41.544696 23073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:41.546942 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.537656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38109 (local address 127.20.199.62:46695)
0925 14:53:41.537733 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.537736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.537917 (+ 181us) server_negotiation.cc:407] Connection header received
0925 14:53:41.538378 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.538380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.538397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.538482 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.539144 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.539981 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.541652 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.541979 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.542054 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.544162 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.544164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.544164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.544172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.545740 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.546147 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.546148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.546148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.546210 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.546449 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.546449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.546449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.546546 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.546594 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.546818 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.546880 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:53:41.547201 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.537611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.20.199.1:38109)
0925 14:53:41.537762 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.537800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.538277 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.538562 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.538566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.538866 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:41.539045 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.539049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.540085 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.540087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.541543 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.541546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.542075 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.542076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.542198 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.542683 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:41.542691 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:41.544070 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:41.545809 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.545814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.545815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.546084 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.546267 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.546268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.546268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.546369 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.546599 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:41.546604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:41.546710 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.546791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.546950 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260925 14:53:41.547756 23204 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46695
I20260925 14:53:41.547835 23204 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:41.548004 23204 heartbeater.cc:511] Master 127.20.199.62:46695 requested a full tablet report, sending...
I20260925 14:53:41.548542 23001 ts_manager.cc:194] Registered new tserver with Master: 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845)
I20260925 14:53:41.549362 23001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:38109
I20260925 14:53:41.553493 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 23073
I20260925 14:53:41.553591 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:41.553789 21276 external_mini_cluster.cc:1428] Setting key 76d282ad7191a73ba08c678e0d1c4ffe
I20260925 14:53:41.554270 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.548753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.20.199.1:43517)
0925 14:53:41.548918 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.548927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.548973 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.549092 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.549093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.549238 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:41.549288 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.549289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.550106 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.550107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.550606 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.550607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.550926 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.550927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.550982 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.551350 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:41.551355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:41.551658 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:41.553112 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.553112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.553113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.553281 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.553572 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.553573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.553573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.553643 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.553961 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:41.553962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:41.554009 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.554101 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.554204 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260925 14:53:41.554319 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.548814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:43517 (local address 127.20.199.62:46695)
0925 14:53:41.548857 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.548859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.548982 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:53:41.549007 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.549007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.549017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.549081 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.549348 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.550008 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.550677 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.550861 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.550916 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.551738 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.551739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.551739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.551745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.553035 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.553353 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.553354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.553354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.553442 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.553708 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.553709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.553710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.553856 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.553929 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.554150 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.554239 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:41.578425 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46695
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:41.659446 23213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:41.659654 23213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:41.659682 23213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:41.660735 23213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:41.660770 23213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:41.660789 23213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:53:41.660802 23213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:41.660818 23213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:41.662284 23213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:46695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:41.662496 23213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:41.662688 23213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:41.667512 23219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.667570 23218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.667584 23221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:41.668735 23213 server_base.cc:1034] running on GCE node
I20260925 14:53:41.668987 23213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:41.669224 23213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:41.670395 23213 hybrid_clock.cc:648] HybridClock initialized: now 1790348021670368 us; error 56 us; skew 500 ppm
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:41.672894 23213 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:41.673341 23213 webserver.cc:460] Webserver started at http://127.20.199.2:45977/ using document root <none> and password file <none>
I20260925 14:53:41.673573 23213 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:41.673624 23213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:41.673743 23213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:41.674726 23213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "2cc2199fb64949b1827df4f331a077c8"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "9440f0737f178d8b1424959dc4d377c5"
server_key_iv: "416dc02ce86f6cdb63965322f4dfcd07"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.675015 23213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "2cc2199fb64949b1827df4f331a077c8"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "9440f0737f178d8b1424959dc4d377c5"
server_key_iv: "416dc02ce86f6cdb63965322f4dfcd07"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.676292 23213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:41.677021 23228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.677199 23213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:41.677300 23213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "2cc2199fb64949b1827df4f331a077c8"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "9440f0737f178d8b1424959dc4d377c5"
server_key_iv: "416dc02ce86f6cdb63965322f4dfcd07"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.677362 23213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:41.697193 23213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:41.699798 23213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:41.699928 23213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:41.700431 23213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:41.700479 23213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.700505 23213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:41.700568 23213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:41.709096 23213 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44421
I20260925 14:53:41.709200 23343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44421 every 8 connection(s)
I20260925 14:53:41.709578 23213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:41.712704 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.700932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:39573 (local address 127.20.199.62:46695)
0925 14:53:41.701018 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.701020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.701160 (+ 140us) server_negotiation.cc:407] Connection header received
0925 14:53:41.701690 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.701691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.701703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.701747 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.702498 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.703297 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.705186 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.705529 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.705602 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.709502 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.709504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.709504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.709515 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.711212 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.711767 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.711768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.711768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.711831 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.712096 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.712096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.712097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.712204 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.712252 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.712541 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.712643 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:53:41.712786 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.700891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.20.199.2:39573)
0925 14:53:41.701046 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.701084 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.701586 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.701823 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.701827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.702126 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:41.702403 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.702411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.703411 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.703413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.705079 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.705084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.705656 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.705657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.705837 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.706458 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:41.706470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:41.708734 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:41.711302 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.711307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.711309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.711638 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.711916 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.711917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.711917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.712028 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.712256 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:41.712259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:41.712372 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.712459 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.712590 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 14:53:41.713194 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 23213
I20260925 14:53:41.713321 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:41.713565 21276 external_mini_cluster.cc:1428] Setting key be6ada59553da7a13e0ebfb7eef95def
I20260925 14:53:41.714251 23344 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46695
I20260925 14:53:41.714330 23344 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:41.714501 23344 heartbeater.cc:511] Master 127.20.199.62:46695 requested a full tablet report, sending...
I20260925 14:53:41.714886 23001 ts_manager.cc:194] Registered new tserver with Master: 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421)
I20260925 14:53:41.715454 23001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:39573
I20260925 14:53:41.721175 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.714454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.20.199.2:53713)
0925 14:53:41.714715 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.714726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.714779 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.714917 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.714918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.715078 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:41.715131 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.715132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.715886 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.715888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.716516 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.716518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.717006 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.717007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.717074 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.717672 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:41.717677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:41.718006 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:41.719949 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.719949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.719950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.720209 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.720439 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.720440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.720440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.720501 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.720865 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:41.720866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:41.720912 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.721005 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.721097 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":62,"threads_started":1}
I20260925 14:53:41.721233 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.714380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:53713 (local address 127.20.199.62:46695)
0925 14:53:41.714420 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.714422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.714769 (+ 347us) server_negotiation.cc:407] Connection header received
0925 14:53:41.714836 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.714837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.714847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.714902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.715207 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.715772 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.716611 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.716914 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.716987 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.718094 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.718095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.718096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.718111 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.719859 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.720284 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.720285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.720285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.720346 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.720577 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.720578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.720578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.720769 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.720846 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.721044 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.721125 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:41.740288 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46695
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:41.819891 23353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:41.820123 23353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:41.820164 23353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:41.821323 23353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:41.821390 23353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:41.821455 23353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:53:41.821521 23353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:53:41.821563 23353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:41.822996 23353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:46695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:41.823257 23353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:41.823491 23353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:41.828513 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.828502 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:41.828502 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:41.828758 23353 server_base.cc:1034] running on GCE node
I20260925 14:53:41.828985 23353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:41.829222 23353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:41.830381 23353 hybrid_clock.cc:648] HybridClock initialized: now 1790348021830357 us; error 53 us; skew 500 ppm
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:41.832840 23353 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:41.833233 23353 webserver.cc:460] Webserver started at http://127.20.199.3:40143/ using document root <none> and password file <none>
I20260925 14:53:41.833508 23353 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:41.833577 23353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:41.833736 23353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:41.834888 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "220090d54eff4bae80d031c568555731"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "f9628e256d5468e3981e497b3c09bc7c"
server_key_iv: "b1d0ff742c7dfd2a0d90a315a5705160"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.835237 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "220090d54eff4bae80d031c568555731"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "f9628e256d5468e3981e497b3c09bc7c"
server_key_iv: "b1d0ff742c7dfd2a0d90a315a5705160"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.836521 23353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:41.837273 23368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.837453 23353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:41.837555 23353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "220090d54eff4bae80d031c568555731"
format_stamp: "Formatted at 2026-09-25 14:53:41 on dist-test-slave-8qt6"
server_key: "f9628e256d5468e3981e497b3c09bc7c"
server_key_iv: "b1d0ff742c7dfd2a0d90a315a5705160"
server_key_version: "encryptionkey@0"
I20260925 14:53:41.837638 23353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:41.849179 23353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:41.851384 23353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:41.851544 23353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:41.852030 23353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:41.852103 23353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.852149 23353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:41.852178 23353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:41.858794 23353 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:43205
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:41.858867 23483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:43205 every 8 connection(s)
I20260925 14:53:41.859136 23353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:41.861941 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.852490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:53477 (local address 127.20.199.62:46695)
0925 14:53:41.852560 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.852562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.852766 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:53:41.853264 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.853266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.853287 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.853342 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.854298 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.855005 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.856517 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.856864 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.856947 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.859361 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.859363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.859363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.859371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.860576 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.860977 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.860978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.860978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.861035 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.861290 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.861291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.861291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.861448 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.861510 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.861767 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.861853 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:53:41.862106 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.852466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.20.199.3:53477)
0925 14:53:41.852632 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.852670 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.853154 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.853474 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.853479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.853877 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:41.854182 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.854190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.855096 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.855098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.856417 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.856420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.856963 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.856964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.857085 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.857590 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:41.857600 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:41.859263 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:41.860674 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.860680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.860682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.860895 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.861110 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.861111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.861111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.861216 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.861502 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:41.861508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:41.861638 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.861742 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.861889 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260925 14:53:41.862629 23484 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46695
I20260925 14:53:41.862705 23484 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:41.862869 23484 heartbeater.cc:511] Master 127.20.199.62:46695 requested a full tablet report, sending...
I20260925 14:53:41.863292 23000 ts_manager.cc:194] Registered new tserver with Master: 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:41.863780 23000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:53477
I20260925 14:53:41.865105 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 23353
I20260925 14:53:41.865232 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:41.865450 21276 external_mini_cluster.cc:1428] Setting key d348a40f477e42c9b234635116239656
I20260925 14:53:41.866412 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:41.869801 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.863554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.20.199.3:52625)
0925 14:53:41.863705 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.863714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.863770 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.864126 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.864127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.864341 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:41.864403 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.864404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.865060 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.865061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.865949 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.865952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.866432 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.866433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.866507 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.866856 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:41.866861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:41.867232 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:41.868716 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.868717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.868717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.868950 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.869146 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:41.869146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:41.869147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:41.869216 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:41.869466 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:41.869467 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:41.869516 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.869589 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.869718 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260925 14:53:41.869977 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.863638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:52625 (local address 127.20.199.62:46695)
0925 14:53:41.863690 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.863692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.863761 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:53:41.863894 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.863895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.863906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.863974 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.864484 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.864955 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.866086 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.866371 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.866429 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.867314 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.867315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.867315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.867321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.868637 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.869024 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.869025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.869025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.869075 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.869277 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.869278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.869278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.869368 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.869470 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.869628 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.869731 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:53:41 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:41.892066 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.884777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.20.199.62:46695)
0925 14:53:41.884860 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.884862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.884872 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:41.884897 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.884898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.884910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.884976 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:41.885816 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.886451 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.887192 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.887399 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.887490 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.889818 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:41.889819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:41.889819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:41.889825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:41.890996 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.891394 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.891395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.891395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.891450 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:41.891659 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:41.891659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:41.891660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:41.891758 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:41.891830 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.891914 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.891971 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:53:41.893795 23000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:41.895309 23000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:41.902083 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.897957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44421 (local address 127.0.0.1:34682)
0925 14:53:41.898422 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.898430 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.898471 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.898677 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.898678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.898680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:41.898716 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.898718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.899578 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.899579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.900741 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.900744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.901764 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.901765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.901858 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.901871 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.901938 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.901991 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260925 14:53:41.902199 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.897762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.0.0.1:44708)
0925 14:53:41.898180 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.898222 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.898309 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.898576 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.898579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.898582 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:41.898644 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.898646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.899977 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.899978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.900915 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.900918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.901935 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.901936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.901994 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.901997 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.902048 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.902079 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":59,"threads_started":1}
I20260925 14:53:41.902283 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.897953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.20.199.1:37845)
0925 14:53:41.898119 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.898125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.898316 (+ 191us) server_negotiation.cc:407] Connection header received
0925 14:53:41.898399 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.898401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.898420 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.898498 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:41.898730 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.899877 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.901020 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.901778 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.901819 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.901876 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.902095 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.902154 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260925 14:53:41.902446 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.898079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.20.199.2:44421)
0925 14:53:41.898246 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.898248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.898524 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:53:41.898565 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.898567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.898583 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.898684 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:41.898774 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.899491 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.900825 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.901603 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.901673 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.901727 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.902287 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.902334 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260925 14:53:41.902848 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.897762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:43205 (local address 127.0.0.1:48306)
0925 14:53:41.898250 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:41.898258 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:41.898367 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:41.898751 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:41.898752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:41.898755 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:41.898810 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.898812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.900022 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.900023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.901092 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:41.901095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.902621 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.902622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:41.902694 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.902699 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:41.902739 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 14:53:41.902777 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260925 14:53:41.903013 23138 tablet_service.cc:1467] Processing CreateTablet for tablet 2e12fc239b3e4860a213b350c72abe11 (DEFAULT_TABLE table=test-table [id=4907234d552141b989fb120312396cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:41.903092 23278 tablet_service.cc:1467] Processing CreateTablet for tablet 2e12fc239b3e4860a213b350c72abe11 (DEFAULT_TABLE table=test-table [id=4907234d552141b989fb120312396cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:41.903240 23503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:41.897850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.20.199.3:43205)
0925 14:53:41.898115 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:41.898120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:41.898308 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:53:41.898534 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:41.898537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:41.898556 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:41.898696 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:41.898875 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.899938 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.901186 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:41.902074 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:41.902999 (+ 925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:41.903065 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:41.903091 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 14:53:41.903144 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260925 14:53:41.903415 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e12fc239b3e4860a213b350c72abe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:41.903447 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e12fc239b3e4860a213b350c72abe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:41.906037 23418 tablet_service.cc:1467] Processing CreateTablet for tablet 2e12fc239b3e4860a213b350c72abe11 (DEFAULT_TABLE table=test-table [id=4907234d552141b989fb120312396cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:41.906389 23418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e12fc239b3e4860a213b350c72abe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:41.906397 23507 tablet_bootstrap.cc:492] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Bootstrap starting.
I20260925 14:53:41.906684 23508 tablet_bootstrap.cc:492] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Bootstrap starting.
I20260925 14:53:41.907174 23507 tablet_bootstrap.cc:654] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:41.907280 23508 tablet_bootstrap.cc:654] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:41.907534 23507 log.cc:824] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:41.907588 23508 log.cc:824] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:41.908310 23507 tablet_bootstrap.cc:492] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: No bootstrap required, opened a new log
I20260925 14:53:41.908310 23508 tablet_bootstrap.cc:492] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: No bootstrap required, opened a new log
I20260925 14:53:41.908380 23508 ts_tablet_manager.cc:1397] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:41.908380 23507 ts_tablet_manager.cc:1397] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:41.909317 23511 tablet_bootstrap.cc:492] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Bootstrap starting.
I20260925 14:53:41.910113 23511 tablet_bootstrap.cc:654] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:41.910419 23511 log.cc:824] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:41.910538 23508 raft_consensus.cc:348] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:41.910719 23508 raft_consensus.cc:374] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:41.910761 23508 raft_consensus.cc:729] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfd072c467641189d2fd36b9dff47cc, State: Initialized, Role: FOLLOWER
I20260925 14:53:41.910671 23507 raft_consensus.cc:348] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:41.910801 23507 raft_consensus.cc:374] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:41.910837 23507 raft_consensus.cc:729] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Initialized, Role: FOLLOWER
I20260925 14:53:41.910899 23508 consensus_queue.cc:260] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:41.910933 23507 consensus_queue.cc:260] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:41.911152 23511 tablet_bootstrap.cc:492] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: No bootstrap required, opened a new log
I20260925 14:53:41.911209 23507 ts_tablet_manager.cc:1428] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:41.911222 23508 ts_tablet_manager.cc:1428] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:41.911229 23511 ts_tablet_manager.cc:1397] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:41.911355 23344 heartbeater.cc:503] Master 127.20.199.62:46695 was elected leader, sending a full tablet report...
I20260925 14:53:41.911444 23204 heartbeater.cc:503] Master 127.20.199.62:46695 was elected leader, sending a full tablet report...
I20260925 14:53:41.913082 23511 raft_consensus.cc:348] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:41.913264 23511 raft_consensus.cc:374] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:41.913313 23511 raft_consensus.cc:729] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220090d54eff4bae80d031c568555731, State: Initialized, Role: FOLLOWER
I20260925 14:53:41.913456 23511 consensus_queue.cc:260] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:41.913730 23511 ts_tablet_manager.cc:1428] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:53:41.913784 23484 heartbeater.cc:503] Master 127.20.199.62:46695 was elected leader, sending a full tablet report...
W20260925 14:53:41.960214 23345 tablet.cc:2369] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:42.017562 23513 raft_consensus.cc:482] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:42.017762 23513 raft_consensus.cc:504] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:42.018186 23513 leader_election.cc:288] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220090d54eff4bae80d031c568555731 (127.20.199.3:43205), 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845)
I20260925 14:53:42.021209 23503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.018496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:59379 (local address 127.20.199.3:43205)
0925 14:53:42.018540 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.018543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.018553 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:42.018585 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.018586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.018596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.018656 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:42.018838 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.019371 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.020481 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.020833 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.020886 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.020893 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.021088 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.021135 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:53:42.021238 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.018371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:43205 (local address 127.20.199.2:59379)
0925 14:53:42.018430 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.018443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.018496 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.018678 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.018679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.018681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:42.018760 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.018762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.019457 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.019458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.020376 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.020379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.020915 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.020917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.021008 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.021013 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.021075 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.021122 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:53:42.021256 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.018419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:50583 (local address 127.20.199.1:37845)
0925 14:53:42.018475 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.018477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.018601 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:53:42.018688 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.018689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.018701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.018767 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:42.018959 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.019462 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.020464 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.020893 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.020971 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.020982 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.021150 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.021193 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:42.021267 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.018374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.20.199.2:50583)
0925 14:53:42.018535 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.018546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.018610 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.018786 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.018787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.018788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:42.018891 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.018893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.019549 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.019550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.020377 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.020379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.020975 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.020976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.021066 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.021070 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.021127 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.021163 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":12,"thread_start_us":81,"threads_started":1}
I20260925 14:53:42.021643 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e12fc239b3e4860a213b350c72abe11" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc" is_pre_election: true
I20260925 14:53:42.021656 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e12fc239b3e4860a213b350c72abe11" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731" is_pre_election: true
I20260925 14:53:42.021800 23438 raft_consensus.cc:2454] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 0.
I20260925 14:53:42.021816 23158 raft_consensus.cc:2454] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 0.
I20260925 14:53:42.022082 23232 leader_election.cc:302] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2199fb64949b1827df4f331a077c8, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:42.022217 23513 raft_consensus.cc:2781] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:42.022279 23513 raft_consensus.cc:482] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:42.022306 23513 raft_consensus.cc:3037] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.023242 23513 raft_consensus.cc:504] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:42.023386 23513 leader_election.cc:288] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Requested vote from peers 220090d54eff4bae80d031c568555731 (127.20.199.3:43205), 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845)
I20260925 14:53:42.023531 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e12fc239b3e4860a213b350c72abe11" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc"
I20260925 14:53:42.023521 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e12fc239b3e4860a213b350c72abe11" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731"
I20260925 14:53:42.023612 23158 raft_consensus.cc:3037] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.023628 23438 raft_consensus.cc:3037] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.024312 23158 raft_consensus.cc:2454] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 1.
I20260925 14:53:42.024456 23438 raft_consensus.cc:2454] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 1.
I20260925 14:53:42.024462 23232 leader_election.cc:302] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2199fb64949b1827df4f331a077c8, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:42.024679 23513 raft_consensus.cc:2781] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:42.024824 23513 raft_consensus.cc:686] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Running, Role: LEADER
I20260925 14:53:42.024895 23513 consensus_queue.cc:237] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } }
I20260925 14:53:42.025799 23000 catalog_manager.cc:5495] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "2cc2199fb64949b1827df4f331a077c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } health_report { overall_health: HEALTHY } } }
W20260925 14:53:42.045451 23205 tablet.cc:2369] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:53:42 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:42.068876 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.062216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.20.199.62:46695)
0925 14:53:42.062282 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.062284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.062479 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:53:42.062536 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.062537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.062547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.062630 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:42.063060 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.063576 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.064234 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.064454 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.064509 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.066565 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:42.066566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:42.066566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:42.066572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:42.067945 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:42.068279 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:42.068280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:42.068280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:42.068326 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:42.068491 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:42.068492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:42.068492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:42.068585 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:42.068633 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.068749 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.068813 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:53:42.069043 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.062186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46695 (local address 127.0.0.1:36612)
0925 14:53:42.062378 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.062391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.062472 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.062635 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.062636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.062921 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:42.062966 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.062968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.063641 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.063642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.064150 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.064151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.064520 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.064520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.064560 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.065091 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:42.065096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:42.066489 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:42.068015 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:42.068020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:42.068022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:42.068206 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:42.068380 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:42.068381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:42.068381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:42.068436 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:42.068634 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:42.068640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:42.068685 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.068736 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.068879 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":60,"thread_start_us":119,"threads_started":1}
I20260925 14:53:42.070837 23000 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:36612:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:53:42.071026 23000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:42.072414 23138 tablet_service.cc:1467] Processing CreateTablet for tablet 5b369edcdd674b6c9525c1451913080d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f0f7c0062f843aeb5b751fd959b6d2d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:42.072459 23278 tablet_service.cc:1467] Processing CreateTablet for tablet 5b369edcdd674b6c9525c1451913080d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f0f7c0062f843aeb5b751fd959b6d2d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:42.072525 23418 tablet_service.cc:1467] Processing CreateTablet for tablet 5b369edcdd674b6c9525c1451913080d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f0f7c0062f843aeb5b751fd959b6d2d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:53:42.072621 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b369edcdd674b6c9525c1451913080d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:42.072621 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b369edcdd674b6c9525c1451913080d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:42.072676 23418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b369edcdd674b6c9525c1451913080d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:42.073952 23508 tablet_bootstrap.cc:492] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: Bootstrap starting.
I20260925 14:53:42.074123 23511 tablet_bootstrap.cc:492] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: Bootstrap starting.
I20260925 14:53:42.074136 23507 tablet_bootstrap.cc:492] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: Bootstrap starting.
I20260925 14:53:42.074366 23508 tablet_bootstrap.cc:654] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:42.074581 23507 tablet_bootstrap.cc:654] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:42.074715 23511 tablet_bootstrap.cc:654] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:42.075114 23508 tablet_bootstrap.cc:492] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: No bootstrap required, opened a new log
I20260925 14:53:42.075182 23508 ts_tablet_manager.cc:1397] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:42.075351 23511 tablet_bootstrap.cc:492] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: No bootstrap required, opened a new log
I20260925 14:53:42.075351 23507 tablet_bootstrap.cc:492] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: No bootstrap required, opened a new log
I20260925 14:53:42.075403 23507 ts_tablet_manager.cc:1397] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:42.075403 23511 ts_tablet_manager.cc:1397] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:42.075350 23508 raft_consensus.cc:348] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.075435 23508 raft_consensus.cc:374] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:42.075461 23508 raft_consensus.cc:729] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfd072c467641189d2fd36b9dff47cc, State: Initialized, Role: FOLLOWER
I20260925 14:53:42.075502 23508 consensus_queue.cc:260] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.075548 23511 raft_consensus.cc:348] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.075541 23507 raft_consensus.cc:348] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.075611 23508 ts_tablet_manager.cc:1428] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:42.075618 23507 raft_consensus.cc:374] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:42.075618 23511 raft_consensus.cc:374] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:42.075644 23507 raft_consensus.cc:729] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Initialized, Role: FOLLOWER
I20260925 14:53:42.075644 23511 raft_consensus.cc:729] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220090d54eff4bae80d031c568555731, State: Initialized, Role: FOLLOWER
I20260925 14:53:42.075685 23511 consensus_queue.cc:260] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.075685 23507 consensus_queue.cc:260] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.075767 23507 ts_tablet_manager.cc:1428] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:42.075780 23511 ts_tablet_manager.cc:1428] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:42.075886 23514 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.075966 23514 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:42.075908 23515 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.075968 23513 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.076012 23515 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:42.076041 23513 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:53:42.109769 23485 tablet.cc:2369] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:42.186244 23514 raft_consensus.cc:482] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:42.186342 23514 raft_consensus.cc:504] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.186620 23514 leader_election.cc:288] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421), 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:42.189863 23503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.187073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58497 (local address 127.20.199.3:43205)
0925 14:53:42.187116 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.187118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.187238 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:53:42.187281 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.187282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.187291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.187347 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:42.187513 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.187977 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.189043 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.189487 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.189522 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.189528 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.189764 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.189801 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:42.189883 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.187004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:43205 (local address 127.20.199.1:58497)
0925 14:53:42.187150 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.187160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.187225 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.187348 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.187349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.187367 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:42.187451 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.187453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.188057 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.188057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.188916 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.188918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.189612 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.189613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.189681 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.189685 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.189746 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.189786 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":20,"thread_start_us":52,"threads_started":1}
I20260925 14:53:42.190034 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.186900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:60687 (local address 127.20.199.2:44421)
0925 14:53:42.186970 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.186973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.187044 (+ 71us) server_negotiation.cc:407] Connection header received
0925 14:53:42.187168 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.187170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.187182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.187238 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:42.187391 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.187912 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.189032 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.189467 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.189504 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.189511 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.189934 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.189973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:53:42.190048 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.186806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44421 (local address 127.20.199.1:60687)
0925 14:53:42.186956 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.186970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.187065 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.187256 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.187257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.187259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:42.187320 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.187322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.187985 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.187986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.188937 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.188939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.189804 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.189805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.189851 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.189854 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.189925 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.189964 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 14:53:42.190481 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b369edcdd674b6c9525c1451913080d" candidate_uuid: "3bfd072c467641189d2fd36b9dff47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731" is_pre_election: true
I20260925 14:53:42.190599 23438 raft_consensus.cc:2454] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bfd072c467641189d2fd36b9dff47cc in term 0.
I20260925 14:53:42.190577 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b369edcdd674b6c9525c1451913080d" candidate_uuid: "3bfd072c467641189d2fd36b9dff47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc2199fb64949b1827df4f331a077c8" is_pre_election: true
I20260925 14:53:42.190701 23298 raft_consensus.cc:2454] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bfd072c467641189d2fd36b9dff47cc in term 0.
I20260925 14:53:42.190779 23089 leader_election.cc:302] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220090d54eff4bae80d031c568555731, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:42.190943 23514 raft_consensus.cc:2781] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:42.190991 23514 raft_consensus.cc:482] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:42.191012 23514 raft_consensus.cc:3037] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.191710 23514 raft_consensus.cc:504] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.191843 23514 leader_election.cc:288] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [CANDIDATE]: Term 1 election: Requested vote from peers 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421), 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:42.191999 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b369edcdd674b6c9525c1451913080d" candidate_uuid: "3bfd072c467641189d2fd36b9dff47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc2199fb64949b1827df4f331a077c8"
I20260925 14:53:42.192068 23298 raft_consensus.cc:3037] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.192041 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b369edcdd674b6c9525c1451913080d" candidate_uuid: "3bfd072c467641189d2fd36b9dff47cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731"
I20260925 14:53:42.192122 23438 raft_consensus.cc:3037] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.192699 23298 raft_consensus.cc:2454] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bfd072c467641189d2fd36b9dff47cc in term 1.
I20260925 14:53:42.192762 23438 raft_consensus.cc:2454] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bfd072c467641189d2fd36b9dff47cc in term 1.
I20260925 14:53:42.192867 23089 leader_election.cc:302] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2199fb64949b1827df4f331a077c8, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:42.192951 23514 raft_consensus.cc:2781] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:42.193074 23514 raft_consensus.cc:686] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [term 1 LEADER]: Becoming Leader. State: Replica: 3bfd072c467641189d2fd36b9dff47cc, State: Running, Role: LEADER
I20260925 14:53:42.193161 23514 consensus_queue.cc:237] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.193619 23530 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: TxnStatusTablet state changed. Reason: New leader 3bfd072c467641189d2fd36b9dff47cc. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.193699 23530 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:42.193934 23532 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:53:42.194037 23532 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:53:42.193969 23000 catalog_manager.cc:5495] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1). New cstate: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:42.223133 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.220201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.0.0.1:44712)
0925 14:53:42.220338 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.220351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.220413 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.220598 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.220599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.220600 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:42.220653 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.220655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.221238 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.221239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.222219 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.222223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.222552 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.222554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.222669 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.222687 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.222947 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.222958 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.223016 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.223066 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 14:53:42.223124 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.220260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.20.199.1:37845)
0925 14:53:42.220330 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.220332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.220391 (+ 59us) server_negotiation.cc:407] Connection header received
0925 14:53:42.220502 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.220502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.220519 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.220576 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:42.220714 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.221170 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.222303 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.222483 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.222538 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.222784 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.222878 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.222965 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.223022 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.223043 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:53:42.225061 23298 raft_consensus.cc:1261] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Refusing update from remote peer 3bfd072c467641189d2fd36b9dff47cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:42.225090 23438 raft_consensus.cc:1261] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Refusing update from remote peer 3bfd072c467641189d2fd36b9dff47cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:42.225380 23514 consensus_queue.cc:1035] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:42.225566 23530 consensus_queue.cc:1035] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:42.226560 23513 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: TxnStatusTablet state changed. Reason: New leader 3bfd072c467641189d2fd36b9dff47cc. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.226686 23513 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:42.226868 23515 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: TxnStatusTablet state changed. Reason: New leader 3bfd072c467641189d2fd36b9dff47cc. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.226981 23515 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:42.227692 23514 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.227780 23514 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:42.227843 23535 mvcc.cc:205] Tried to move back new op lower bound from 7333265499031236608 to 7333265498903724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333265499031236608}]
I20260925 14:53:42.228351 23513 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.228431 23513 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 2cc2199fb64949b1827df4f331a077c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:42.228739 23514 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.228812 23514 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 3bfd072c467641189d2fd36b9dff47cc: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:53:42.229211 23515 tablet_replica.cc:439] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bfd072c467641189d2fd36b9dff47cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } } }
I20260925 14:53:42.229306 23515 tablet_replica.cc:442] T 5b369edcdd674b6c9525c1451913080d P 220090d54eff4bae80d031c568555731: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:53:42.229352 23537 mvcc.cc:205] Tried to move back new op lower bound from 7333265499031236608 to 7333265498903724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333265499031236608}]
I20260925 14:53:42.232889 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.230421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.20.199.2:44421)
0925 14:53:42.230459 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.230461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.230471 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:42.230558 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.230559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.230583 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.230660 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:42.230814 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.231310 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.232086 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.232307 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.232384 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.232556 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.232658 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.232749 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.232792 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.232812 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:42.237815 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.235276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57995 (local address 127.20.199.1:37845)
0925 14:53:42.235314 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.235316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.235463 (+ 147us) server_negotiation.cc:407] Connection header received
0925 14:53:42.235514 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.235515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.235522 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.235567 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:42.235693 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.236114 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.237184 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.237357 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.237420 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.237618 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.237641 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.237680 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.237754 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.237769 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:42.237880 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.235223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.20.199.2:57995)
0925 14:53:42.235386 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.235398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.235458 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.235591 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.235592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.235594 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:42.235636 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.235637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.236175 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.236175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.237098 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.237101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.237437 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.237438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.237502 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.237515 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.237698 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.237700 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.237753 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.237795 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260925 14:53:42.240394 23438 raft_consensus.cc:1261] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Refusing update from remote peer 2cc2199fb64949b1827df4f331a077c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:42.240507 23158 raft_consensus.cc:1261] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Refusing update from remote peer 2cc2199fb64949b1827df4f331a077c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:42.240636 23517 consensus_queue.cc:1035] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:42.240732 23513 consensus_queue.cc:1035] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:42.242708 23536 mvcc.cc:205] Tried to move back new op lower bound from 7333265499094958080 to 7333265498214666240. Current Snapshot: MvccSnapshot[applied={T|T < 7333265499094958080}]
I20260925 14:53:42.250406 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.247544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:40037 (local address 127.20.199.2:44421)
0925 14:53:42.247583 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:42.247586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:42.247730 (+ 144us) server_negotiation.cc:407] Connection header received
0925 14:53:42.247817 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:42.247817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:42.247848 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:42.247904 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:42.248040 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.248548 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.249799 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.249973 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.250025 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.250216 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.250240 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.250287 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:42.250344 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:53:42.250359 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:42.250465 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:42.247507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44421 (local address 127.20.199.1:40037)
0925 14:53:42.247669 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:42.247680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:42.247756 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:42.247915 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:42.247915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:42.247917 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:42.247982 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.247984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.248616 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.248617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.249723 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:42.249726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:42.250049 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:42.250050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:42.250127 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:42.250141 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:42.250294 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:42.250295 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:53:42.250343 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:53:42.250381 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260925 14:53:42.259397 23001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:42.259505 23001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:42.260300 23001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Sending DeleteTablet for 3 replicas of tablet 2e12fc239b3e4860a213b350c72abe11
I20260925 14:53:42.260695 23138 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e12fc239b3e4860a213b350c72abe11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:42 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44708
I20260925 14:53:42.260701 23278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e12fc239b3e4860a213b350c72abe11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:42 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:34682
I20260925 14:53:42.261009 23555 tablet_replica.cc:330] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: stopping tablet replica
I20260925 14:53:42.261130 23555 raft_consensus.cc:2229] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:42.261130 23001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:42.261278 23001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:42.261214 23555 raft_consensus.cc:2258] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:42.261868 23555 ts_tablet_manager.cc:1905] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:42.262437 23138 tablet_service.cc:1467] Processing CreateTablet for tablet 81332559ffdf415e925937ab3222f350 (DEFAULT_TABLE table=test-table [id=edbda46c70b440b09795dd23f1163e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:42.262560 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81332559ffdf415e925937ab3222f350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:42.263616 23555 ts_tablet_manager.cc:1918] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:42.263742 23555 log.cc:1196] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/2e12fc239b3e4860a213b350c72abe11
I20260925 14:53:42.263803 23508 tablet_bootstrap.cc:492] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Bootstrap starting.
I20260925 14:53:42.264066 23508 tablet_bootstrap.cc:654] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:42.264178 23555 ts_tablet_manager.cc:1939] T 2e12fc239b3e4860a213b350c72abe11 P 3bfd072c467641189d2fd36b9dff47cc: Deleting consensus metadata
I20260925 14:53:42.264626 22988 catalog_manager.cc:4841] TS 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845): tablet 2e12fc239b3e4860a213b350c72abe11 (table test-table [id=4907234d552141b989fb120312396cdd]) successfully deleted
I20260925 14:53:42.264745 23278 tablet_service.cc:1467] Processing CreateTablet for tablet 81332559ffdf415e925937ab3222f350 (DEFAULT_TABLE table=test-table [id=edbda46c70b440b09795dd23f1163e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:42.264853 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81332559ffdf415e925937ab3222f350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:42.265775 23508 tablet_bootstrap.cc:492] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: No bootstrap required, opened a new log
I20260925 14:53:42.265821 23508 ts_tablet_manager.cc:1397] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:42.265944 23508 raft_consensus.cc:348] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.266000 23508 raft_consensus.cc:374] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:42.266021 23508 raft_consensus.cc:729] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfd072c467641189d2fd36b9dff47cc, State: Initialized, Role: FOLLOWER
I20260925 14:53:42.266062 23508 consensus_queue.cc:260] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.266125 23507 tablet_bootstrap.cc:492] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Bootstrap starting.
I20260925 14:53:42.266155 23508 ts_tablet_manager.cc:1428] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:42.266530 23507 tablet_bootstrap.cc:654] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:42.267556 23507 tablet_bootstrap.cc:492] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: No bootstrap required, opened a new log
I20260925 14:53:42.267607 23507 ts_tablet_manager.cc:1397] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:42.267738 23507 raft_consensus.cc:348] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.267808 23507 raft_consensus.cc:374] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:42.267830 23507 raft_consensus.cc:729] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Initialized, Role: FOLLOWER
I20260925 14:53:42.267869 23507 consensus_queue.cc:260] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.267953 23507 ts_tablet_manager.cc:1428] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:42.268396 23418 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e12fc239b3e4860a213b350c72abe11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:42 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260925 14:53:42.268628 23417 tablet_service.cc:1467] Processing CreateTablet for tablet 81332559ffdf415e925937ab3222f350 (DEFAULT_TABLE table=test-table [id=edbda46c70b440b09795dd23f1163e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:42.268760 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81332559ffdf415e925937ab3222f350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:42.269526 23559 tablet_replica.cc:330] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: stopping tablet replica
I20260925 14:53:42.269672 23556 tablet_replica.cc:330] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: stopping tablet replica
I20260925 14:53:42.269798 23556 raft_consensus.cc:2229] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:42.269925 23559 raft_consensus.cc:2229] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:42.270081 23556 raft_consensus.cc:2258] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:42.270183 23559 raft_consensus.cc:2258] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:42.270548 23511 tablet_bootstrap.cc:492] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Bootstrap starting.
I20260925 14:53:42.270771 23556 ts_tablet_manager.cc:1905] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:42.270913 23511 tablet_bootstrap.cc:654] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:42.271946 23559 ts_tablet_manager.cc:1905] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:42.272339 23513 raft_consensus.cc:482] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:42.272395 23513 raft_consensus.cc:504] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.272516 23513 leader_election.cc:288] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845), 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:42.272711 23556 ts_tablet_manager.cc:1918] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:42.272732 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81332559ffdf415e925937ab3222f350" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc" is_pre_election: true
I20260925 14:53:42.272807 23158 raft_consensus.cc:2454] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 0.
I20260925 14:53:42.272931 23232 leader_election.cc:302] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2199fb64949b1827df4f331a077c8, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:42.273039 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81332559ffdf415e925937ab3222f350" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731" is_pre_election: true
I20260925 14:53:42.272918 23556 log.cc:1196] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/2e12fc239b3e4860a213b350c72abe11
I20260925 14:53:42.273192 23513 raft_consensus.cc:2781] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:42.273237 23513 raft_consensus.cc:482] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:42.273279 23513 raft_consensus.cc:3037] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.273559 23556 ts_tablet_manager.cc:1939] T 2e12fc239b3e4860a213b350c72abe11 P 2cc2199fb64949b1827df4f331a077c8: Deleting consensus metadata
I20260925 14:53:42.273921 23513 raft_consensus.cc:504] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.274031 23513 leader_election.cc:288] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Requested vote from peers 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845), 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:42.274039 22985 catalog_manager.cc:4841] TS 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421): tablet 2e12fc239b3e4860a213b350c72abe11 (table test-table [id=4907234d552141b989fb120312396cdd]) successfully deleted
I20260925 14:53:42.274242 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81332559ffdf415e925937ab3222f350" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc"
I20260925 14:53:42.274322 23158 raft_consensus.cc:3037] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.274955 23158 raft_consensus.cc:2454] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 1.
I20260925 14:53:42.275113 23232 leader_election.cc:302] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2199fb64949b1827df4f331a077c8, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:42.275406 23513 raft_consensus.cc:2781] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:42.275460 23513 raft_consensus.cc:686] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Running, Role: LEADER
I20260925 14:53:42.275497 23513 consensus_queue.cc:237] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
W20260925 14:53:42.275870 23229 leader_election.cc:341] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 220090d54eff4bae80d031c568555731 (127.20.199.3:43205): Illegal state: must be running to vote when last-logged opid is not known
I20260925 14:53:42.276036 23001 catalog_manager.cc:5495] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "2cc2199fb64949b1827df4f331a077c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:42.276571 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81332559ffdf415e925937ab3222f350" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731"
W20260925 14:53:42.276873 23229 leader_election.cc:341] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 220090d54eff4bae80d031c568555731 (127.20.199.3:43205): Illegal state: must be running to vote when last-logged opid is not known
I20260925 14:53:42.278029 23511 tablet_bootstrap.cc:492] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: No bootstrap required, opened a new log
I20260925 14:53:42.278080 23511 ts_tablet_manager.cc:1397] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 14:53:42.278220 23511 raft_consensus.cc:348] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.278281 23511 raft_consensus.cc:374] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:42.278301 23511 raft_consensus.cc:729] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220090d54eff4bae80d031c568555731, State: Initialized, Role: FOLLOWER
I20260925 14:53:42.278342 23511 consensus_queue.cc:260] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:42.278420 23511 ts_tablet_manager.cc:1428] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:42.279767 23559 ts_tablet_manager.cc:1918] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:42.279835 23559 log.cc:1196] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/2e12fc239b3e4860a213b350c72abe11
I20260925 14:53:42.282219 23559 ts_tablet_manager.cc:1939] T 2e12fc239b3e4860a213b350c72abe11 P 220090d54eff4bae80d031c568555731: Deleting consensus metadata
I20260925 14:53:42.282627 23158 raft_consensus.cc:1261] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Refusing update from remote peer 2cc2199fb64949b1827df4f331a077c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:42.282696 22985 catalog_manager.cc:4841] TS 220090d54eff4bae80d031c568555731 (127.20.199.3:43205): tablet 2e12fc239b3e4860a213b350c72abe11 (table test-table [id=4907234d552141b989fb120312396cdd]) successfully deleted
I20260925 14:53:42.282750 23438 raft_consensus.cc:3037] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:42.282819 23550 consensus_queue.cc:1035] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:42.283474 23438 raft_consensus.cc:1261] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Refusing update from remote peer 2cc2199fb64949b1827df4f331a077c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:42.283696 23550 consensus_queue.cc:1035] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:53:42.288689 23536 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:53:43.806699 23001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:43.806799 23001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:43.807590 23001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Sending DeleteTablet for 3 replicas of tablet 81332559ffdf415e925937ab3222f350
I20260925 14:53:43.807886 23138 tablet_service.cc:1514] Processing DeleteTablet for tablet 81332559ffdf415e925937ab3222f350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44708
I20260925 14:53:43.807888 23417 tablet_service.cc:1514] Processing DeleteTablet for tablet 81332559ffdf415e925937ab3222f350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260925 14:53:43.808072 23278 tablet_service.cc:1514] Processing DeleteTablet for tablet 81332559ffdf415e925937ab3222f350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:34682
I20260925 14:53:43.808261 23588 tablet_replica.cc:330] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: stopping tablet replica
I20260925 14:53:43.808316 23588 raft_consensus.cc:2229] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:43.808372 23588 raft_consensus.cc:2258] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.809701 23588 ts_tablet_manager.cc:1905] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.809960 23586 tablet_replica.cc:330] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: stopping tablet replica
I20260925 14:53:43.810070 23586 raft_consensus.cc:2229] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.810166 23586 raft_consensus.cc:2258] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.810447 23586 ts_tablet_manager.cc:1905] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.811437 23588 ts_tablet_manager.cc:1918] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:53:43.811497 23588 log.cc:1196] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/81332559ffdf415e925937ab3222f350
I20260925 14:53:43.811755 23588 ts_tablet_manager.cc:1939] T 81332559ffdf415e925937ab3222f350 P 2cc2199fb64949b1827df4f331a077c8: Deleting consensus metadata
I20260925 14:53:43.811895 23587 tablet_replica.cc:330] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: stopping tablet replica
I20260925 14:53:43.811968 23587 raft_consensus.cc:2229] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.811980 23586 ts_tablet_manager.cc:1918] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:53:43.812014 23587 raft_consensus.cc:2258] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.812026 23586 log.cc:1196] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/81332559ffdf415e925937ab3222f350
I20260925 14:53:43.812245 23586 ts_tablet_manager.cc:1939] T 81332559ffdf415e925937ab3222f350 P 3bfd072c467641189d2fd36b9dff47cc: Deleting consensus metadata
I20260925 14:53:43.812270 23587 ts_tablet_manager.cc:1905] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.812866 22988 catalog_manager.cc:4841] TS 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845): tablet 81332559ffdf415e925937ab3222f350 (table test-table [id=edbda46c70b440b09795dd23f1163e32]) successfully deleted
I20260925 14:53:43.812908 22985 catalog_manager.cc:4841] TS 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421): tablet 81332559ffdf415e925937ab3222f350 (table test-table [id=edbda46c70b440b09795dd23f1163e32]) successfully deleted
I20260925 14:53:43.813807 23587 ts_tablet_manager.cc:1918] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:53:43.813884 23587 log.cc:1196] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/81332559ffdf415e925937ab3222f350
I20260925 14:53:43.814091 23587 ts_tablet_manager.cc:1939] T 81332559ffdf415e925937ab3222f350 P 220090d54eff4bae80d031c568555731: Deleting consensus metadata
I20260925 14:53:43.814747 22985 catalog_manager.cc:4841] TS 220090d54eff4bae80d031c568555731 (127.20.199.3:43205): tablet 81332559ffdf415e925937ab3222f350 (table test-table [id=edbda46c70b440b09795dd23f1163e32]) successfully deleted
Sep 25 14:53:43 dist-test-slave-8qt6 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:53:43 dist-test-slave-8qt6 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:43.832796 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.825378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.20.199.62:46695)
0925 14:53:43.825545 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.825547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.825590 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:53:43.825714 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.825716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.825733 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.825813 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.826402 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.826913 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.827764 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.827979 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.828056 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.830463 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.830464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.830464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.830472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.831810 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.832158 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.832158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.832159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.832214 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.832378 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.832378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.832378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.832478 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.832555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.832657 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.832724 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 14:53:43.834431 23001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:43.834592 23001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:43.835865 23138 tablet_service.cc:1467] Processing CreateTablet for tablet 2488a779674c4e9ea6f16f6212eafd34 (DEFAULT_TABLE table=test-table [id=91733695f3b64acd9fe7a6cb4336c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.835942 23417 tablet_service.cc:1467] Processing CreateTablet for tablet 2488a779674c4e9ea6f16f6212eafd34 (DEFAULT_TABLE table=test-table [id=91733695f3b64acd9fe7a6cb4336c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.836012 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2488a779674c4e9ea6f16f6212eafd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.836050 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2488a779674c4e9ea6f16f6212eafd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.836050 23278 tablet_service.cc:1467] Processing CreateTablet for tablet 2488a779674c4e9ea6f16f6212eafd34 (DEFAULT_TABLE table=test-table [id=91733695f3b64acd9fe7a6cb4336c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.836238 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2488a779674c4e9ea6f16f6212eafd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.837577 23598 tablet_bootstrap.cc:492] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Bootstrap starting.
I20260925 14:53:43.837970 23598 tablet_bootstrap.cc:654] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.838212 23599 tablet_bootstrap.cc:492] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Bootstrap starting.
I20260925 14:53:43.838310 23597 tablet_bootstrap.cc:492] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Bootstrap starting.
I20260925 14:53:43.838608 23599 tablet_bootstrap.cc:654] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.838676 23597 tablet_bootstrap.cc:654] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.839192 23599 tablet_bootstrap.cc:492] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: No bootstrap required, opened a new log
I20260925 14:53:43.839239 23597 tablet_bootstrap.cc:492] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: No bootstrap required, opened a new log
I20260925 14:53:43.839253 23599 ts_tablet_manager.cc:1397] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:43.839279 23597 ts_tablet_manager.cc:1397] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:43.839380 23599 raft_consensus.cc:348] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.839380 23597 raft_consensus.cc:348] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.839452 23597 raft_consensus.cc:374] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.839452 23599 raft_consensus.cc:374] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.839475 23597 raft_consensus.cc:729] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfd072c467641189d2fd36b9dff47cc, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.839475 23599 raft_consensus.cc:729] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.839541 23597 consensus_queue.cc:260] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.839550 23599 consensus_queue.cc:260] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.839648 23597 ts_tablet_manager.cc:1428] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.839648 23599 ts_tablet_manager.cc:1428] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:43.839844 23598 tablet_bootstrap.cc:492] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: No bootstrap required, opened a new log
I20260925 14:53:43.839891 23598 ts_tablet_manager.cc:1397] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:43.840013 23598 raft_consensus.cc:348] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.840097 23598 raft_consensus.cc:374] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.840128 23598 raft_consensus.cc:729] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220090d54eff4bae80d031c568555731, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.840169 23598 consensus_queue.cc:260] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.840327 23598 ts_tablet_manager.cc:1428] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:43.931731 23603 raft_consensus.cc:482] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:43.931847 23603 raft_consensus.cc:504] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.932245 23603 leader_election.cc:288] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845), 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421)
I20260925 14:53:43.936709 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.932390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.20.199.3:47791)
0925 14:53:43.932673 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.932687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.932751 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.933009 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.933010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.933013 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:43.933079 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.933081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.934177 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.934178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.935516 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.935518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.936371 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.936372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.936512 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.936517 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.936570 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.936624 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":47,"thread_start_us":115,"threads_started":1}
I20260925 14:53:43.936728 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.932496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:47791 (local address 127.20.199.1:37845)
0925 14:53:43.932687 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.932690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.932814 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:53:43.932844 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.932845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.932857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.932924 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:43.933162 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.934072 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.935618 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.936286 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.936374 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.936384 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.936575 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.936640 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260925 14:53:43.936913 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.932691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:47599 (local address 127.20.199.2:44421)
0925 14:53:43.932860 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.932862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.932872 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:43.932903 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.932903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.932915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.932999 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:43.933183 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.934105 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.935622 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.936337 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.936430 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.936442 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.936748 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.936815 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260925 14:53:43.937131 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.932448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44421 (local address 127.20.199.3:47599)
0925 14:53:43.932765 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.932775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.932843 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.933036 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.933036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.933038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:43.933072 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.933074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.934200 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.934201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.935516 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.935518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.936484 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.936486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.936579 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.936584 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.936638 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.936691 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":61,"thread_start_us":122,"threads_started":1}
I20260925 14:53:43.937451 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2488a779674c4e9ea6f16f6212eafd34" candidate_uuid: "220090d54eff4bae80d031c568555731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc" is_pre_election: true
I20260925 14:53:43.937563 23158 raft_consensus.cc:2454] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220090d54eff4bae80d031c568555731 in term 0.
I20260925 14:53:43.937518 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2488a779674c4e9ea6f16f6212eafd34" candidate_uuid: "220090d54eff4bae80d031c568555731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc2199fb64949b1827df4f331a077c8" is_pre_election: true
I20260925 14:53:43.937633 23298 raft_consensus.cc:2454] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220090d54eff4bae80d031c568555731 in term 0.
I20260925 14:53:43.937759 23372 leader_election.cc:302] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220090d54eff4bae80d031c568555731, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:43.937894 23603 raft_consensus.cc:2781] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:43.937930 23603 raft_consensus.cc:482] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:43.937958 23603 raft_consensus.cc:3037] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:43.938977 23603 raft_consensus.cc:504] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.939127 23603 leader_election.cc:288] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [CANDIDATE]: Term 1 election: Requested vote from peers 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845), 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421)
I20260925 14:53:43.939320 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2488a779674c4e9ea6f16f6212eafd34" candidate_uuid: "220090d54eff4bae80d031c568555731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc"
I20260925 14:53:43.939329 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2488a779674c4e9ea6f16f6212eafd34" candidate_uuid: "220090d54eff4bae80d031c568555731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc2199fb64949b1827df4f331a077c8"
I20260925 14:53:43.939395 23298 raft_consensus.cc:3037] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:43.939396 23158 raft_consensus.cc:3037] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:43.940156 23298 raft_consensus.cc:2454] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220090d54eff4bae80d031c568555731 in term 1.
I20260925 14:53:43.940173 23158 raft_consensus.cc:2454] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220090d54eff4bae80d031c568555731 in term 1.
I20260925 14:53:43.940338 23372 leader_election.cc:302] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220090d54eff4bae80d031c568555731, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:43.940431 23603 raft_consensus.cc:2781] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:43.940577 23603 raft_consensus.cc:686] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 1 LEADER]: Becoming Leader. State: Replica: 220090d54eff4bae80d031c568555731, State: Running, Role: LEADER
I20260925 14:53:43.940661 23603 consensus_queue.cc:237] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.941380 23000 catalog_manager.cc:5495] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220090d54eff4bae80d031c568555731 (127.20.199.3). New cstate: current_term: 1 leader_uuid: "220090d54eff4bae80d031c568555731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } health_report { overall_health: HEALTHY } } }
I20260925 14:53:43.959581 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.957142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.20.199.3:43205)
0925 14:53:43.957264 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.957266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.957271 (+ 5us) server_negotiation.cc:407] Connection header received
0925 14:53:43.957304 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.957304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.957323 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.957386 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:43.957607 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.958104 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.958846 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.959042 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.959108 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.959250 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.959353 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:43.959407 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.959453 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.959468 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":78,"threads_started":1}
I20260925 14:53:43.964259 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.961892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:42491 (local address 127.20.199.1:37845)
0925 14:53:43.961931 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.961933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.962085 (+ 152us) server_negotiation.cc:407] Connection header received
0925 14:53:43.962114 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.962115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.962128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.962186 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:43.962288 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.962829 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.963643 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.963863 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.963918 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.964069 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.964094 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:43.964132 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.964187 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.964204 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:43.964283 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.961815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.20.199.3:42491)
0925 14:53:43.961990 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.961999 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.962051 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.962181 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.962182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.962183 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:43.962242 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.962243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.962901 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.962902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.963569 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.963571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.963933 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.963934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.963980 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.963992 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.964147 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:43.964148 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.964179 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.964217 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260925 14:53:43.966756 23298 raft_consensus.cc:1261] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Refusing update from remote peer 220090d54eff4bae80d031c568555731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:43.966777 23158 raft_consensus.cc:1261] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Refusing update from remote peer 220090d54eff4bae80d031c568555731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:43.967020 23608 consensus_queue.cc:1035] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:43.967141 23603 consensus_queue.cc:1035] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:43.975085 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.972600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58185 (local address 127.20.199.3:43205)
0925 14:53:43.972716 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.972718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.972726 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:43.972747 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.972748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.972758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.972813 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:43.972903 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.973542 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.974396 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.974734 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.974785 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.974905 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.974931 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:43.974968 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.975016 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.975033 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:53:43.975103 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.972442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:43205 (local address 127.20.199.1:58185)
0925 14:53:43.972599 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.972611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.972663 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.972822 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.972823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.972824 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:53:43.972863 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.972864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.973612 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.973612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.974313 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.974316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.974796 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.974797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.974839 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.974843 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.974980 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:53:43.974981 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.975015 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.975048 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260925 14:53:43.981559 23000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:43.981653 23000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:43.982306 23000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Sending DeleteTablet for 3 replicas of tablet 2488a779674c4e9ea6f16f6212eafd34
I20260925 14:53:43.982558 23278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2488a779674c4e9ea6f16f6212eafd34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:34682
I20260925 14:53:43.982575 23138 tablet_service.cc:1514] Processing DeleteTablet for tablet 2488a779674c4e9ea6f16f6212eafd34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44708
I20260925 14:53:43.982573 23417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2488a779674c4e9ea6f16f6212eafd34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260925 14:53:43.982648 23588 tablet_replica.cc:330] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: stopping tablet replica
I20260925 14:53:43.982659 23587 tablet_replica.cc:330] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: stopping tablet replica
I20260925 14:53:43.982659 23586 tablet_replica.cc:330] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: stopping tablet replica
I20260925 14:53:43.982702 23587 raft_consensus.cc:2229] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:43.982707 23586 raft_consensus.cc:2229] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.982715 23588 raft_consensus.cc:2229] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.982744 23586 raft_consensus.cc:2258] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.982750 23588 raft_consensus.cc:2258] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.982789 23587 raft_consensus.cc:2258] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.983062 23586 ts_tablet_manager.cc:1905] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.983170 23588 ts_tablet_manager.cc:1905] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.984589 23000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 14:53:43.984727 23587 ts_tablet_manager.cc:1905] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:53:43.984748 23000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:43.985039 23588 ts_tablet_manager.cc:1918] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:43.985110 23588 log.cc:1196] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/2488a779674c4e9ea6f16f6212eafd34
I20260925 14:53:43.985322 23588 ts_tablet_manager.cc:1939] T 2488a779674c4e9ea6f16f6212eafd34 P 2cc2199fb64949b1827df4f331a077c8: Deleting consensus metadata
I20260925 14:53:43.985671 22985 catalog_manager.cc:4841] TS 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421): tablet 2488a779674c4e9ea6f16f6212eafd34 (table test-table [id=91733695f3b64acd9fe7a6cb4336c647]) successfully deleted
I20260925 14:53:43.986366 23587 ts_tablet_manager.cc:1918] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:43.986426 23587 log.cc:1196] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/2488a779674c4e9ea6f16f6212eafd34
I20260925 14:53:43.986615 23278 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed44668724436ca05da070a0151ad0 (DEFAULT_TABLE table=test-table [id=ee0c8742824745a6a3c04dc574a3ea54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.986634 23417 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed44668724436ca05da070a0151ad0 (DEFAULT_TABLE table=test-table [id=ee0c8742824745a6a3c04dc574a3ea54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.986658 23587 ts_tablet_manager.cc:1939] T 2488a779674c4e9ea6f16f6212eafd34 P 220090d54eff4bae80d031c568555731: Deleting consensus metadata
I20260925 14:53:43.986727 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed44668724436ca05da070a0151ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.986752 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed44668724436ca05da070a0151ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.986900 23138 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed44668724436ca05da070a0151ad0 (DEFAULT_TABLE table=test-table [id=ee0c8742824745a6a3c04dc574a3ea54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.987027 23586 ts_tablet_manager.cc:1918] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:53:43.987063 23586 log.cc:1196] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/2488a779674c4e9ea6f16f6212eafd34
I20260925 14:53:43.987264 23586 ts_tablet_manager.cc:1939] T 2488a779674c4e9ea6f16f6212eafd34 P 3bfd072c467641189d2fd36b9dff47cc: Deleting consensus metadata
I20260925 14:53:43.987390 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed44668724436ca05da070a0151ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.987883 23598 tablet_bootstrap.cc:492] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Bootstrap starting.
I20260925 14:53:43.988291 23598 tablet_bootstrap.cc:654] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.988654 23599 tablet_bootstrap.cc:492] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Bootstrap starting.
I20260925 14:53:43.988755 22985 catalog_manager.cc:4841] TS 220090d54eff4bae80d031c568555731 (127.20.199.3:43205): tablet 2488a779674c4e9ea6f16f6212eafd34 (table test-table [id=91733695f3b64acd9fe7a6cb4336c647]) successfully deleted
I20260925 14:53:43.988962 23599 tablet_bootstrap.cc:654] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.989040 23598 tablet_bootstrap.cc:492] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: No bootstrap required, opened a new log
I20260925 14:53:43.989104 23598 ts_tablet_manager.cc:1397] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:43.989231 23597 tablet_bootstrap.cc:492] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc: Bootstrap starting.
I20260925 14:53:43.989274 23598 raft_consensus.cc:348] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.989360 23598 raft_consensus.cc:374] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.989383 23598 raft_consensus.cc:729] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220090d54eff4bae80d031c568555731, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.989445 23598 consensus_queue.cc:260] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.989534 23597 tablet_bootstrap.cc:654] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.989579 23598 ts_tablet_manager.cc:1428] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:43.989984 23599 tablet_bootstrap.cc:492] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: No bootstrap required, opened a new log
I20260925 14:53:43.990029 23599 ts_tablet_manager.cc:1397] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:43.990120 23599 raft_consensus.cc:348] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.990211 23599 raft_consensus.cc:374] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.990235 23599 raft_consensus.cc:729] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.990285 23599 consensus_queue.cc:260] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.990388 23599 ts_tablet_manager.cc:1428] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.990607 23597 tablet_bootstrap.cc:492] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc: No bootstrap required, opened a new log
I20260925 14:53:43.990649 23597 ts_tablet_manager.cc:1397] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:43.990792 23597 raft_consensus.cc:348] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.990859 23597 raft_consensus.cc:374] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.990880 23597 raft_consensus.cc:729] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfd072c467641189d2fd36b9dff47cc, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.990923 23597 consensus_queue.cc:260] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:43.991014 23597 ts_tablet_manager.cc:1428] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.991309 22988 catalog_manager.cc:4841] TS 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845): tablet 2488a779674c4e9ea6f16f6212eafd34 (table test-table [id=91733695f3b64acd9fe7a6cb4336c647]) successfully deleted
I20260925 14:53:44.232167 23565 raft_consensus.cc:482] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:44.232285 23565 raft_consensus.cc:504] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:44.232508 23565 leader_election.cc:288] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845), 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:44.232757 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed44668724436ca05da070a0151ad0" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc" is_pre_election: true
I20260925 14:53:44.232784 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed44668724436ca05da070a0151ad0" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731" is_pre_election: true
I20260925 14:53:44.232862 23158 raft_consensus.cc:2454] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 0.
I20260925 14:53:44.232895 23438 raft_consensus.cc:2454] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 0.
I20260925 14:53:44.233026 23232 leader_election.cc:302] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2199fb64949b1827df4f331a077c8, 3bfd072c467641189d2fd36b9dff47cc; no voters:
I20260925 14:53:44.233141 23565 raft_consensus.cc:2781] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:44.233209 23565 raft_consensus.cc:482] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:44.233240 23565 raft_consensus.cc:3037] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:44.233980 23565 raft_consensus.cc:504] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:44.234129 23565 leader_election.cc:288] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Requested vote from peers 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845), 220090d54eff4bae80d031c568555731 (127.20.199.3:43205)
I20260925 14:53:44.234292 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed44668724436ca05da070a0151ad0" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220090d54eff4bae80d031c568555731"
I20260925 14:53:44.234292 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed44668724436ca05da070a0151ad0" candidate_uuid: "2cc2199fb64949b1827df4f331a077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfd072c467641189d2fd36b9dff47cc"
I20260925 14:53:44.234372 23438 raft_consensus.cc:3037] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:44.234375 23158 raft_consensus.cc:3037] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:44.235121 23438 raft_consensus.cc:2454] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 1.
I20260925 14:53:44.235122 23158 raft_consensus.cc:2454] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc2199fb64949b1827df4f331a077c8 in term 1.
I20260925 14:53:44.235330 23229 leader_election.cc:302] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220090d54eff4bae80d031c568555731, 2cc2199fb64949b1827df4f331a077c8; no voters:
I20260925 14:53:44.235415 23565 raft_consensus.cc:2781] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:44.235471 23565 raft_consensus.cc:686] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc2199fb64949b1827df4f331a077c8, State: Running, Role: LEADER
I20260925 14:53:44.235524 23565 consensus_queue.cc:237] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } }
I20260925 14:53:44.236133 23001 catalog_manager.cc:5495] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "2cc2199fb64949b1827df4f331a077c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc2199fb64949b1827df4f331a077c8" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:44.277931 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:44.275918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.20.199.2:44421)
0925 14:53:44.275972 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:44.275975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:44.275984 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:44.276046 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:44.276047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:44.276057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:44.276093 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:44.276172 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:44.276628 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:44.277316 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:44.277562 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:44.277611 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:44.277741 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:44.277766 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:44.277804 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:44.277856 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:53:44.277874 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:53:44.280076 23438 raft_consensus.cc:1261] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Refusing update from remote peer 2cc2199fb64949b1827df4f331a077c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:44.280100 23158 raft_consensus.cc:1261] T e5ed44668724436ca05da070a0151ad0 P 3bfd072c467641189d2fd36b9dff47cc [term 1 FOLLOWER]: Refusing update from remote peer 2cc2199fb64949b1827df4f331a077c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:44.280292 23565 consensus_queue.cc:1035] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220090d54eff4bae80d031c568555731" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:44.280375 23580 consensus_queue.cc:1035] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfd072c467641189d2fd36b9dff47cc" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:53:44.284523 23569 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:53:45.801040 23001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:45.801160 23001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:45.801949 23001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23675b867b914828b042926bd23db129: Sending DeleteTablet for 3 replicas of tablet e5ed44668724436ca05da070a0151ad0
I20260925 14:53:45.802219 23138 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed44668724436ca05da070a0151ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:45 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44708
I20260925 14:53:45.802310 23417 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed44668724436ca05da070a0151ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:45 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260925 14:53:45.802336 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 23073
I20260925 14:53:45.802470 23278 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed44668724436ca05da070a0151ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:45 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:34682
I20260925 14:53:45.803354 23652 tablet_replica.cc:330] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: stopping tablet replica
I20260925 14:53:45.803416 23652 raft_consensus.cc:2229] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:45.803455 23652 raft_consensus.cc:2258] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:45.803721 23652 ts_tablet_manager.cc:1905] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:45.803915 23653 tablet_replica.cc:330] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: stopping tablet replica
I20260925 14:53:45.803980 23653 raft_consensus.cc:2229] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:45.804049 23653 raft_consensus.cc:2258] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:45.804268 23653 ts_tablet_manager.cc:1905] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:45.805383 23652 ts_tablet_manager.cc:1918] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:53:45.805496 23652 log.cc:1196] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/e5ed44668724436ca05da070a0151ad0
I20260925 14:53:45.805713 23652 ts_tablet_manager.cc:1939] T e5ed44668724436ca05da070a0151ad0 P 220090d54eff4bae80d031c568555731: Deleting consensus metadata
I20260925 14:53:45.806025 23653 ts_tablet_manager.cc:1918] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:53:45.806089 23653 log.cc:1196] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/e5ed44668724436ca05da070a0151ad0
I20260925 14:53:45.806098 22985 catalog_manager.cc:4841] TS 220090d54eff4bae80d031c568555731 (127.20.199.3:43205): tablet e5ed44668724436ca05da070a0151ad0 (table test-table [id=ee0c8742824745a6a3c04dc574a3ea54]) successfully deleted
I20260925 14:53:45.806325 23653 ts_tablet_manager.cc:1939] T e5ed44668724436ca05da070a0151ad0 P 2cc2199fb64949b1827df4f331a077c8: Deleting consensus metadata
I20260925 14:53:45.806648 22985 catalog_manager.cc:4841] TS 2cc2199fb64949b1827df4f331a077c8 (127.20.199.2:44421): tablet e5ed44668724436ca05da070a0151ad0 (table test-table [id=ee0c8742824745a6a3c04dc574a3ea54]) successfully deleted
W20260925 14:53:45.808806 22988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:37845 (error 108)
I20260925 14:53:45.808992 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 23213
I20260925 14:53:45.809906 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:45.809675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:37845 (local address 127.0.0.1:47906)
0925 14:53:45.809806 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:45.809856 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
W20260925 14:53:45.810060 22988 catalog_manager.cc:4584] TS 3bfd072c467641189d2fd36b9dff47cc (127.20.199.1:37845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5ed44668724436ca05da070a0151ad0: Network error: Client connection negotiation failed: client connection to 127.20.199.1:37845: connect: Connection refused (error 111)
I20260925 14:53:45.815615 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 23353
I20260925 14:53:45.820889 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 22969
2026-09-25T14:53:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5718 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:45 dist-test-slave-8qt6 krb5kdc[23658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:45 dist-test-slave-8qt6 krb5kdc[23658](info): set up 2 sockets
Sep 25 14:53:45 dist-test-slave-8qt6 krb5kdc[23658](info): commencing operation
krb5kdc: starting...
W20260925 14:53:46.957046 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:47.073338 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46855
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:47.154867 23671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:47.155074 23671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:47.155098 23671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:47.156253 23671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:47.156302 23671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:47.156318 23671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:47.156383 23671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:47.156415 23671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:47.157943 23671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:47.158203 23671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:47.158452 23671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:47.164090 23671 server_base.cc:1034] running on GCE node
W20260925 14:53:47.164144 23676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.164143 23677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.164198 23679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:47.164546 23671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:47.164839 23671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:47.166019 23671 hybrid_clock.cc:648] HybridClock initialized: now 1790348027166006 us; error 30 us; skew 500 ppm
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:47.168548 23671 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:47.169034 23671 webserver.cc:460] Webserver started at http://127.20.199.62:45685/ using document root <none> and password file <none>
I20260925 14:53:47.169281 23671 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:47.169353 23671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:47.169545 23671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:47.170627 23671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "12cda19e3dbe42cab36e28848fbaca5c"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "dc5b3f3727c6d1f0b911813a1141622d"
server_key_iv: "beb76b330afc16f4ee55042103e157fd"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.171018 23671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "12cda19e3dbe42cab36e28848fbaca5c"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "dc5b3f3727c6d1f0b911813a1141622d"
server_key_iv: "beb76b330afc16f4ee55042103e157fd"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.172451 23671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:47.173216 23686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.173522 23671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:47.173638 23671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "12cda19e3dbe42cab36e28848fbaca5c"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "dc5b3f3727c6d1f0b911813a1141622d"
server_key_iv: "beb76b330afc16f4ee55042103e157fd"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.173719 23671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:47.217478 23671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:47.219807 23671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:47.219978 23671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:47.224077 23671 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:46855
I20260925 14:53:47.224116 23738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:46855 every 8 connection(s)
I20260925 14:53:47.224467 23671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:47.225468 23739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:47.227660 23739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c: Bootstrap starting.
I20260925 14:53:47.228255 23739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:47.228338 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 23671
I20260925 14:53:47.228431 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:47.228627 21276 external_mini_cluster.cc:1428] Setting key f671151d0decfbda933bab103b6b4807
I20260925 14:53:47.228609 23739 log.cc:824] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:47.229636 23739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c: No bootstrap required, opened a new log
I20260925 14:53:47.231909 23739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cda19e3dbe42cab36e28848fbaca5c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46855 } }
I20260925 14:53:47.232082 23739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:47.232120 23739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cda19e3dbe42cab36e28848fbaca5c, State: Initialized, Role: FOLLOWER
I20260925 14:53:47.232241 23739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cda19e3dbe42cab36e28848fbaca5c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46855 } }
I20260925 14:53:47.232311 23739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:47.232337 23739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:47.232378 23739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:47.232986 23739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cda19e3dbe42cab36e28848fbaca5c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46855 } }
I20260925 14:53:47.233111 23739 leader_election.cc:302] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12cda19e3dbe42cab36e28848fbaca5c; no voters:
I20260925 14:53:47.233299 23739 leader_election.cc:288] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:47.233331 23744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:47.233568 23744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [term 1 LEADER]: Becoming Leader. State: Replica: 12cda19e3dbe42cab36e28848fbaca5c, State: Running, Role: LEADER
I20260925 14:53:47.233652 23739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:47.233711 23744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cda19e3dbe42cab36e28848fbaca5c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46855 } }
I20260925 14:53:47.234077 23744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12cda19e3dbe42cab36e28848fbaca5c. Latest consensus state: current_term: 1 leader_uuid: "12cda19e3dbe42cab36e28848fbaca5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cda19e3dbe42cab36e28848fbaca5c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46855 } } }
I20260925 14:53:47.234225 23744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:47.234679 23752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:47.235136 23752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:47.235280 23745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12cda19e3dbe42cab36e28848fbaca5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cda19e3dbe42cab36e28848fbaca5c" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46855 } } }
I20260925 14:53:47.235445 23745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c [sys.catalog]: This master's current role is: LEADER
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:47.236717 23752 catalog_manager.cc:1347] Generated new cluster ID: ffa821ff5713442db552523dc5f58155
I20260925 14:53:47.236773 23752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:47.239668 23752 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:47.240161 23752 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:47.240386 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.229154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.20.199.62:46855)
0925 14:53:47.229307 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:47.229315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:47.229341 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:53:47.229859 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:47.229866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:47.229915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:47.230035 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:47.230796 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.231683 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.232729 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.233160 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.233287 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.236345 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:47.236349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:47.236349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:47.236359 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:47.238370 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.238794 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.238797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.238797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.238899 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.239106 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.239107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.239107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.239366 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:47.239522 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:47.240006 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 14:53:47.240173 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260925 14:53:47.244575 23752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12cda19e3dbe42cab36e28848fbaca5c: Generated new TSK 0
I20260925 14:53:47.244732 23752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:47.276192 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46855
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:47.378126 23765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:47.378365 23765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:47.378407 23765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:47.379601 23765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:47.379664 23765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:47.379709 23765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:47.381165 23765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:47.381474 23765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:47.381723 23765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:47.386785 23770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.386799 23771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.386799 23773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:47.387254 23765 server_base.cc:1034] running on GCE node
I20260925 14:53:47.387449 23765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:47.387722 23765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:47.388890 23765 hybrid_clock.cc:648] HybridClock initialized: now 1790348027388875 us; error 34 us; skew 500 ppm
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:47.391628 23765 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:47.392167 23765 webserver.cc:460] Webserver started at http://127.20.199.1:33107/ using document root <none> and password file <none>
I20260925 14:53:47.392422 23765 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:47.392490 23765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:47.392628 23765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:47.393776 23765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "8ca7035bbabd49c4a0611232ece178c0"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "875af120f3a43ce2c961d4cf4325adce"
server_key_iv: "4eb0b746dc72df3f90b1743be69bbb40"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.394129 23765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "8ca7035bbabd49c4a0611232ece178c0"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "875af120f3a43ce2c961d4cf4325adce"
server_key_iv: "4eb0b746dc72df3f90b1743be69bbb40"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.395520 23765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:47.396266 23780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.396456 23765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:47.396556 23765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "8ca7035bbabd49c4a0611232ece178c0"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "875af120f3a43ce2c961d4cf4325adce"
server_key_iv: "4eb0b746dc72df3f90b1743be69bbb40"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.396628 23765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:47.417055 23765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:47.419101 23765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:47.419265 23765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:47.419541 23765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:47.419852 23765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:47.419906 23765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.419945 23765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:47.419983 23765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.425825 23765 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:39509
I20260925 14:53:47.425882 23893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:39509 every 8 connection(s)
I20260925 14:53:47.426189 23765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:47.431910 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 23765
I20260925 14:53:47.432022 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:47.432188 21276 external_mini_cluster.cc:1428] Setting key ad70db0ad98e16c8e34bfee5690f87e4
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:47.436311 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.426919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:50855 (local address 127.20.199.62:46855)
0925 14:53:47.426967 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:47.426969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:47.427218 (+ 249us) server_negotiation.cc:407] Connection header received
0925 14:53:47.427699 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:47.427701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:47.427712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:47.427775 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:47.428733 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.429247 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.430247 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.430472 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.430539 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.432967 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:47.432968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:47.432968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:47.432977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:47.434775 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.435251 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.435252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.435252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.435322 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.435642 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.435643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.435643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.435743 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:47.435795 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:47.436155 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 14:53:47.436242 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:53:47.436563 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.426897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46855 (local address 127.20.199.1:50855)
0925 14:53:47.427056 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:47.427097 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:47.427591 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:47.427849 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:47.427853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:47.428432 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:47.428622 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:47.428627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.429337 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.429337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:47.430165 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:47.430167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.430572 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.430574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:47.430684 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.431252 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:47.431261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:47.432875 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:47.434849 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:47.434855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:47.434858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:47.435169 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:47.435427 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:47.435428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:47.435428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:47.435575 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:47.435811 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:47.435817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:47.435984 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 14:53:47.436100 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:53:47.436274 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260925 14:53:47.437122 23894 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46855
I20260925 14:53:47.437213 23894 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:47.437480 23894 heartbeater.cc:511] Master 127.20.199.62:46855 requested a full tablet report, sending...
I20260925 14:53:47.438166 23703 ts_manager.cc:194] Registered new tserver with Master: 8ca7035bbabd49c4a0611232ece178c0 (127.20.199.1:39509)
I20260925 14:53:47.438736 23703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:50855
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:47.456207 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46855
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:47.538323 23899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:47.538519 23899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:47.538542 23899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:47.539608 23899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:47.539642 23899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:47.539667 23899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:47.541079 23899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:47.541250 23899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:47.541524 23899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:47.546772 23907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.546816 23904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:47.546831 23899 server_base.cc:1034] running on GCE node
W20260925 14:53:47.546816 23905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:47.547142 23899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:47.547384 23899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:47.548537 23899 hybrid_clock.cc:648] HybridClock initialized: now 1790348027548534 us; error 43 us; skew 500 ppm
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:47.550921 23899 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:47.551378 23899 webserver.cc:460] Webserver started at http://127.20.199.2:32807/ using document root <none> and password file <none>
I20260925 14:53:47.551578 23899 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:47.551618 23899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:47.551700 23899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:47.552659 23899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "941ea1752bb340a6ad71f6298764265a"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "a4100076d871fe779f99c6a87d60980f"
server_key_iv: "96f2ed3952e197b08a3f2480a1e93fb9"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.552956 23899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "941ea1752bb340a6ad71f6298764265a"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "a4100076d871fe779f99c6a87d60980f"
server_key_iv: "96f2ed3952e197b08a3f2480a1e93fb9"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.554296 23899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:47.555011 23914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.555193 23899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.555269 23899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "941ea1752bb340a6ad71f6298764265a"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "a4100076d871fe779f99c6a87d60980f"
server_key_iv: "96f2ed3952e197b08a3f2480a1e93fb9"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.555362 23899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:47.564441 23899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:47.566300 23899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:47.566466 23899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:47.566699 23899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:47.566998 23899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:47.567052 23899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.567093 23899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:47.567121 23899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.573155 23899 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:42103
I20260925 14:53:47.573194 24027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:42103 every 8 connection(s)
I20260925 14:53:47.573601 23899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:47.580940 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 23899
I20260925 14:53:47.581041 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:47.581256 21276 external_mini_cluster.cc:1428] Setting key 8e3a2a5cf25bd45db5b3ec82574ab225
I20260925 14:53:47.582685 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.574326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:48897 (local address 127.20.199.62:46855)
0925 14:53:47.574370 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:47.574372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:47.574578 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:53:47.575024 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:47.575025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:47.575035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:47.575087 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:47.575969 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.576541 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.577591 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.577774 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.577833 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.579762 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:47.579763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:47.579764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:47.579770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:47.581148 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.581690 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.581691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.581692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.581758 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.582030 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.582031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.582031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.582162 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:47.582235 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:47.582514 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:53:47.582597 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:47.582861 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.574315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46855 (local address 127.20.199.2:48897)
0925 14:53:47.574455 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:47.574490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:47.574944 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:47.575143 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:47.575146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:47.575711 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:47.575897 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:47.575902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.576614 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.576614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:47.577534 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:47.577537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.577836 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.577837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:47.577931 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.578353 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:47.578361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:47.579682 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:47.581230 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:47.581235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:47.581237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:47.581606 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:47.581843 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:47.581844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:47.581844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:47.581974 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:47.582238 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:47.582244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:47.582378 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 14:53:47.582492 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:53:47.582639 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260925 14:53:47.583278 24028 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46855
I20260925 14:53:47.583384 24028 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:47.583575 24028 heartbeater.cc:511] Master 127.20.199.62:46855 requested a full tablet report, sending...
I20260925 14:53:47.583951 23703 ts_manager.cc:194] Registered new tserver with Master: 941ea1752bb340a6ad71f6298764265a (127.20.199.2:42103)
I20260925 14:53:47.584326 23703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:48897
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:47.604663 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46855
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:47.685616 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:47.685811 24033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:47.685834 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:47.686983 24033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:47.687024 24033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:47.687050 24033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:47.688458 24033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:47.688685 24033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:47.688899 24033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:47.693780 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.693812 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:47.693984 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:47.693965 24033 server_base.cc:1034] running on GCE node
I20260925 14:53:47.694229 24033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:47.694460 24033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:47.695611 24033 hybrid_clock.cc:648] HybridClock initialized: now 1790348027695593 us; error 36 us; skew 500 ppm
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:47.698026 24033 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:47.698469 24033 webserver.cc:460] Webserver started at http://127.20.199.3:44829/ using document root <none> and password file <none>
I20260925 14:53:47.698714 24033 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:47.698782 24033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:47.698926 24033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:47.700058 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "41b59a6e0de841e1b768e6f636e7a952"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "724cf81cf81bb13a1faf7c94214c8368"
server_key_iv: "376e030d15f25787c3b19d0afdd0eb97"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.700408 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "41b59a6e0de841e1b768e6f636e7a952"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "724cf81cf81bb13a1faf7c94214c8368"
server_key_iv: "376e030d15f25787c3b19d0afdd0eb97"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.701742 24033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:47.702479 24048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.702677 24033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:47.702751 24033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "41b59a6e0de841e1b768e6f636e7a952"
format_stamp: "Formatted at 2026-09-25 14:53:47 on dist-test-slave-8qt6"
server_key: "724cf81cf81bb13a1faf7c94214c8368"
server_key_iv: "376e030d15f25787c3b19d0afdd0eb97"
server_key_version: "encryptionkey@0"
I20260925 14:53:47.702802 24033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:47.715034 24033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:47.717507 24033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:47.717623 24033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:47.717806 24033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:47.718116 24033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:47.718151 24033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.718207 24033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:47.718237 24033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:47.723829 24033 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:41259
I20260925 14:53:47.723904 24161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:41259 every 8 connection(s)
I20260925 14:53:47.724244 24033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:47.729341 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24033
I20260925 14:53:47.729473 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:47.729692 21276 external_mini_cluster.cc:1428] Setting key 5866d236d2319b10358556be0b66a942
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[23658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:47.733285 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.724991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:38149 (local address 127.20.199.62:46855)
0925 14:53:47.725080 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:47.725083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:47.725225 (+ 142us) server_negotiation.cc:407] Connection header received
0925 14:53:47.725767 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:47.725769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:47.725779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:47.725833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:47.726762 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.727252 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.728229 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.728411 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.728466 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.730879 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:47.730880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:47.730881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:47.730887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:47.732092 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.732457 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.732458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.732458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.732506 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:47.732712 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:47.732713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:47.732713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:47.732807 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:47.732857 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:47.733132 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 14:53:47.733210 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:53:47.733495 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:47.724935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46855 (local address 127.20.199.3:38149)
0925 14:53:47.725096 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:47.725131 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:47.725677 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:47.725904 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:47.725907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:47.726490 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:47.726683 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:47.726688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.727334 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.727334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:47.728146 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:47.728148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:47.728498 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:47.728499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:47.728605 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:47.729161 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:47.729171 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:47.730800 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:47.732164 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:47.732170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:47.732171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:47.732391 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:47.732560 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:47.732561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:47.732561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:47.732659 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:47.732858 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:47.732862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:47.732989 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 14:53:47.733087 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:53:47.733264 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260925 14:53:47.733887 24162 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46855
I20260925 14:53:47.733999 24162 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:47.734208 24162 heartbeater.cc:511] Master 127.20.199.62:46855 requested a full tablet report, sending...
I20260925 14:53:47.734522 23703 ts_manager.cc:194] Registered new tserver with Master: 41b59a6e0de841e1b768e6f636e7a952 (127.20.199.3:41259)
I20260925 14:53:47.734961 23703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:38149
I20260925 14:53:47.741387 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:53:47.752112 23743 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:47.751174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.20.199.62:46855)
0925 14:53:47.751255 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:47.751257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:47.751314 (+ 57us) server_negotiation.cc:407] Connection header received
0925 14:53:47.751402 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:47.751403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:47.751412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:47.751467 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:47.752043 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57502: BlockingRecv error: recv got EOF from 127.0.0.1:57502 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:53:47.752993 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 23765
I20260925 14:53:47.757963 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 23899
I20260925 14:53:47.762019 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24033
I20260925 14:53:47.766525 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 23671
2026-09-25T14:53:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1943 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[24174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[24174](info): set up 2 sockets
Sep 25 14:53:47 dist-test-slave-8qt6 krb5kdc[24174](info): commencing operation
krb5kdc: starting...
W20260925 14:53:48.813180 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:48 dist-test-slave-8qt6 krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:48.925042 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38997
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:38997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:49.003495 24187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:49.003728 24187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:49.003763 24187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:49.004907 24187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:49.004979 24187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:49.005005 24187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:49.005026 24187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:49.005055 24187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:49.006567 24187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:38997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:49.006834 24187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:49.007055 24187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:49.011724 24192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:49.011749 24195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:49.011754 24193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:49.012028 24187 server_base.cc:1034] running on GCE node
I20260925 14:53:49.012244 24187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:49.012512 24187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:49.013674 24187 hybrid_clock.cc:648] HybridClock initialized: now 1790348029013670 us; error 45 us; skew 500 ppm
Sep 25 14:53:49 dist-test-slave-8qt6 krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:49.016122 24187 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:49.016594 24187 webserver.cc:460] Webserver started at http://127.20.199.62:37073/ using document root <none> and password file <none>
I20260925 14:53:49.016814 24187 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:49.016888 24187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:49.017037 24187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:49.018136 24187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "d594eb54b8be4cb29c231208c92fd68f"
format_stamp: "Formatted at 2026-09-25 14:53:49 on dist-test-slave-8qt6"
server_key: "0d13616700088fa34e2188ff512b1fdb"
server_key_iv: "286253db5df1a1da8f55644993331d61"
server_key_version: "encryptionkey@0"
I20260925 14:53:49.018499 24187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "d594eb54b8be4cb29c231208c92fd68f"
format_stamp: "Formatted at 2026-09-25 14:53:49 on dist-test-slave-8qt6"
server_key: "0d13616700088fa34e2188ff512b1fdb"
server_key_iv: "286253db5df1a1da8f55644993331d61"
server_key_version: "encryptionkey@0"
I20260925 14:53:49.019773 24187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:49.020509 24202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:49.020677 24187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:49.020780 24187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "d594eb54b8be4cb29c231208c92fd68f"
format_stamp: "Formatted at 2026-09-25 14:53:49 on dist-test-slave-8qt6"
server_key: "0d13616700088fa34e2188ff512b1fdb"
server_key_iv: "286253db5df1a1da8f55644993331d61"
server_key_version: "encryptionkey@0"
I20260925 14:53:49.020854 24187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:49.030514 24187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:49.032164 24187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:49.032330 24187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:49.036305 24187 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:38997
I20260925 14:53:49.036335 24254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:38997 every 8 connection(s)
I20260925 14:53:49.036780 24187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:49.037438 24255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:49.039309 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24187
I20260925 14:53:49.039410 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:49.039587 21276 external_mini_cluster.cc:1428] Setting key 27394b4d2a22a589640ba2d57b0135f1
I20260925 14:53:49.039927 24255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f: Bootstrap starting.
I20260925 14:53:49.040561 24255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:49.040845 24255 log.cc:824] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:49.041553 24255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f: No bootstrap required, opened a new log
I20260925 14:53:49.043782 24255 raft_consensus.cc:348] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d594eb54b8be4cb29c231208c92fd68f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38997 } }
I20260925 14:53:49.043957 24255 raft_consensus.cc:374] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:49.043991 24255 raft_consensus.cc:729] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d594eb54b8be4cb29c231208c92fd68f, State: Initialized, Role: FOLLOWER
I20260925 14:53:49.044101 24255 consensus_queue.cc:260] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d594eb54b8be4cb29c231208c92fd68f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38997 } }
I20260925 14:53:49.044173 24255 raft_consensus.cc:388] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:49.044206 24255 raft_consensus.cc:482] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:49.044250 24255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:49.044926 24255 raft_consensus.cc:504] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d594eb54b8be4cb29c231208c92fd68f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38997 } }
I20260925 14:53:49.045040 24255 leader_election.cc:302] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d594eb54b8be4cb29c231208c92fd68f; no voters:
I20260925 14:53:49.045200 24255 leader_election.cc:288] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:49.045226 24260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:49.045328 24260 raft_consensus.cc:686] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [term 1 LEADER]: Becoming Leader. State: Replica: d594eb54b8be4cb29c231208c92fd68f, State: Running, Role: LEADER
I20260925 14:53:49.045468 24255 sys_catalog.cc:564] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:49.045446 24260 consensus_queue.cc:237] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d594eb54b8be4cb29c231208c92fd68f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38997 } }
I20260925 14:53:49.045801 24260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d594eb54b8be4cb29c231208c92fd68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d594eb54b8be4cb29c231208c92fd68f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38997 } } }
I20260925 14:53:49.045956 24260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:49.046162 24260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d594eb54b8be4cb29c231208c92fd68f. Latest consensus state: current_term: 1 leader_uuid: "d594eb54b8be4cb29c231208c92fd68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d594eb54b8be4cb29c231208c92fd68f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38997 } } }
I20260925 14:53:49.046280 24260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:49.046599 24268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 14:53:49 dist-test-slave-8qt6 krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:49.047094 24268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:49.048705 24268 catalog_manager.cc:1347] Generated new cluster ID: 844e90e288664689a38650e3eb559303
I20260925 14:53:49.048760 24268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:49.051494 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:49.040082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.20.199.62:38997)
0925 14:53:49.040267 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:49.040275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:49.040301 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:53:49.040751 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:49.040758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:49.040820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:49.040969 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:49.041721 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:49.043077 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:49.043876 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:49.044486 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:49.044596 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:49.047584 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:49.047589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:49.047589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:49.047601 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:49.049538 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:49.050040 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:49.050044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:49.050044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:49.050147 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:49.050426 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:49.050427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:49.050427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:49.050714 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:49.050856 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:49.051048 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 14:53:49.051214 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 14:53:49.052501 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:49.055843 24268 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:49.056325 24268 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:49.060429 24268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d594eb54b8be4cb29c231208c92fd68f: Generated new TSK 0
I20260925 14:53:49.060595 24268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:49.076386 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:49.072654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.20.199.62:38997)
0925 14:53:49.072703 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:49.072705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:49.072808 (+ 103us) server_negotiation.cc:407] Connection header received
0925 14:53:49.072845 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:49.072846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:49.072858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:49.072935 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:49.073282 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:49.074128 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:49.074835 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:49.075186 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:49.075257 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:49.075910 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:49.075912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:49.075912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:49.076141 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:49.076161 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:49.076167 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:49.076228 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:49.076277 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:53:49.076302 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:53:49.078702 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24187
2026-09-25T14:53:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1310 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:49 dist-test-slave-8qt6 krb5kdc[24288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:49 dist-test-slave-8qt6 krb5kdc[24288](info): set up 2 sockets
Sep 25 14:53:49 dist-test-slave-8qt6 krb5kdc[24288](info): commencing operation
krb5kdc: starting...
W20260925 14:53:50.109134 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:50.222236 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42353
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:50.300580 24301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:50.300765 24301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:50.300786 24301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:50.301913 24301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:50.301950 24301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:50.301964 24301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:50.301999 24301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:50.302028 24301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:50.303471 24301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:42353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:50.303706 24301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:50.303936 24301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:50.308627 24306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.308640 24307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.308645 24309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:50.308738 24301 server_base.cc:1034] running on GCE node
I20260925 14:53:50.309068 24301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:50.309330 24301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:50.310474 24301 hybrid_clock.cc:648] HybridClock initialized: now 1790348030310471 us; error 40 us; skew 500 ppm
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:50.313004 24301 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:50.313539 24301 webserver.cc:460] Webserver started at http://127.20.199.62:42537/ using document root <none> and password file <none>
I20260925 14:53:50.313813 24301 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:50.313889 24301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:50.314064 24301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:50.315213 24301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "374c6cb8d2cf46de8dfef105e9eee1b5"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "c15fb42fbea1a68abff693388d2714ca"
server_key_iv: "e2594f31d094656b67d2115e9f1e94bb"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.315615 24301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "374c6cb8d2cf46de8dfef105e9eee1b5"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "c15fb42fbea1a68abff693388d2714ca"
server_key_iv: "e2594f31d094656b67d2115e9f1e94bb"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.317168 24301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:50.318003 24316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.318174 24301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.318274 24301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "374c6cb8d2cf46de8dfef105e9eee1b5"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "c15fb42fbea1a68abff693388d2714ca"
server_key_iv: "e2594f31d094656b67d2115e9f1e94bb"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.318346 24301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:50.331357 24301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:50.333129 24301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:50.333288 24301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:50.337304 24301 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:42353
I20260925 14:53:50.337342 24368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:42353 every 8 connection(s)
I20260925 14:53:50.337785 24301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:50.338954 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:50.341331 24369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5: Bootstrap starting.
I20260925 14:53:50.341914 24369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:50.342141 24369 log.cc:824] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:50.342810 24369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5: No bootstrap required, opened a new log
I20260925 14:53:50.344331 24369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42353 } }
I20260925 14:53:50.344463 24369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:50.344484 24369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374c6cb8d2cf46de8dfef105e9eee1b5, State: Initialized, Role: FOLLOWER
I20260925 14:53:50.344580 24369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42353 } }
I20260925 14:53:50.344626 24369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:50.344645 24369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:50.344676 24369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:50.345455 24369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42353 } }
I20260925 14:53:50.345557 24369 leader_election.cc:302] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 374c6cb8d2cf46de8dfef105e9eee1b5; no voters:
I20260925 14:53:50.345675 24369 leader_election.cc:288] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:50.345711 24372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:50.345858 24372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [term 1 LEADER]: Becoming Leader. State: Replica: 374c6cb8d2cf46de8dfef105e9eee1b5, State: Running, Role: LEADER
I20260925 14:53:50.345943 24369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:50.345942 24372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42353 } }
I20260925 14:53:50.346310 24373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42353 } } }
I20260925 14:53:50.346329 24374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 374c6cb8d2cf46de8dfef105e9eee1b5. Latest consensus state: current_term: 1 leader_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374c6cb8d2cf46de8dfef105e9eee1b5" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42353 } } }
I20260925 14:53:50.346395 24373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:50.346401 24374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:50.346622 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24301
I20260925 14:53:50.346722 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:50.346903 21276 external_mini_cluster.cc:1428] Setting key eb759e05948b8ca095dcb912a70d3ee0
I20260925 14:53:50.347378 24380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:50.347905 24380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:50.349372 24380 catalog_manager.cc:1347] Generated new cluster ID: 48382051cfd94bcdbf96c5546b99b8e1
I20260925 14:53:50.349445 24380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:50.357385 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.347568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.20.199.62:42353)
0925 14:53:50.348569 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:50.348578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:50.348613 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:50.348722 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:50.348729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:50.348797 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:50.348947 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:50.349587 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.350468 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.351245 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.351664 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.351794 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.354219 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:50.354222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:50.354223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:50.354234 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:50.355879 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.356256 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.356259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.356260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.356356 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.356560 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.356561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.356562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.356801 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:50.356977 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:50.357045 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:53:50.357183 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":68,"threads_started":1}
I20260925 14:53:50.364490 24380 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:50.365016 24380 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:50.368848 24380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 374c6cb8d2cf46de8dfef105e9eee1b5: Generated new TSK 0
I20260925 14:53:50.369029 24380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:50.400403 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42353
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:50.498165 24395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:50.498384 24395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:50.498421 24395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:50.499545 24395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:50.499606 24395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:50.499650 24395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:50.501547 24395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:50.501767 24395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:50.501935 24395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:50.506801 24401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.506886 24403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.506798 24400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:50.507154 24395 server_base.cc:1034] running on GCE node
I20260925 14:53:50.507333 24395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:50.507577 24395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:50.508728 24395 hybrid_clock.cc:648] HybridClock initialized: now 1790348030508719 us; error 35 us; skew 500 ppm
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:50.511130 24395 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:50.511574 24395 webserver.cc:460] Webserver started at http://127.20.199.1:39191/ using document root <none> and password file <none>
I20260925 14:53:50.511795 24395 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:50.511860 24395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:50.512001 24395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:50.513003 24395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "a4d9e60bb6d141f58697419497d9f911"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "0e74ec85edcce8e525995d3fa61ababc"
server_key_iv: "947c8307ba2b766b10926192a8480961"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.513348 24395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "a4d9e60bb6d141f58697419497d9f911"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "0e74ec85edcce8e525995d3fa61ababc"
server_key_iv: "947c8307ba2b766b10926192a8480961"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.514707 24395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:50.515439 24410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.515619 24395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:50.515717 24395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "a4d9e60bb6d141f58697419497d9f911"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "0e74ec85edcce8e525995d3fa61ababc"
server_key_iv: "947c8307ba2b766b10926192a8480961"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.515785 24395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:50.527730 24395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:50.529613 24395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:50.529765 24395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:50.529970 24395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:50.530259 24395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:50.530309 24395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.530347 24395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:50.530378 24395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.536180 24395 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:35841
I20260925 14:53:50.536228 24523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:35841 every 8 connection(s)
I20260925 14:53:50.536520 24395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:50.544921 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.537201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:42021 (local address 127.20.199.62:42353)
0925 14:53:50.537251 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:50.537253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:50.537458 (+ 205us) server_negotiation.cc:407] Connection header received
0925 14:53:50.537922 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:50.537924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:50.537933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:50.537990 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:50.538840 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.539332 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.540256 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.540458 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.540512 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.542478 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:50.542479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:50.542480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:50.542486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:50.543852 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.544181 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.544181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.544181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.544230 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.544425 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.544425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.544425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.544522 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:50.544572 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:50.544782 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:53:50.544845 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:53:50.545071 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.537158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42353 (local address 127.20.199.1:42021)
0925 14:53:50.537307 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:50.537346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:50.537830 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:50.538039 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:50.538042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:50.538583 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:50.538763 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:50.538768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.539405 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.539406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:50.540177 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:50.540178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.540528 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.540529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:50.540628 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.541056 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:50.541064 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:50.542391 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:50.543918 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:50.543922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:50.543924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:50.544124 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:50.544281 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:50.544281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:50.544281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:50.544374 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:50.544574 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:50.544577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:50.544679 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 14:53:50.544757 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:53:50.544865 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260925 14:53:50.545367 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24395
I20260925 14:53:50.545478 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:50.545583 24524 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42353
I20260925 14:53:50.545675 24524 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:50.545697 21276 external_mini_cluster.cc:1428] Setting key 245ec6afc7e6c2cf0fb377158c309096
I20260925 14:53:50.545855 24524 heartbeater.cc:511] Master 127.20.199.62:42353 requested a full tablet report, sending...
I20260925 14:53:50.546451 24332 ts_manager.cc:194] Registered new tserver with Master: a4d9e60bb6d141f58697419497d9f911 (127.20.199.1:35841)
I20260925 14:53:50.547056 24332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:42021
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:50.568682 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42353
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:50.647994 24529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:50.648180 24529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:50.648203 24529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:50.649380 24529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:50.649452 24529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:50.649502 24529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:50.650906 24529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:50.651139 24529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:50.651353 24529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:50.656157 24534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.656157 24537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.656215 24535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:50.656414 24529 server_base.cc:1034] running on GCE node
I20260925 14:53:50.656596 24529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:50.656785 24529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:50.657922 24529 hybrid_clock.cc:648] HybridClock initialized: now 1790348030657902 us; error 34 us; skew 500 ppm
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:50.660382 24529 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:50.660811 24529 webserver.cc:460] Webserver started at http://127.20.199.2:40439/ using document root <none> and password file <none>
I20260925 14:53:50.661036 24529 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:50.661099 24529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:50.661237 24529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:50.662317 24529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "eb7abd2b2e944cb8943b8c7b5f6b84db"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "dd916787c89ac629e0b0e7e56bb079fc"
server_key_iv: "ec4de9437f00610b50522ff020d3d83d"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.662675 24529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "eb7abd2b2e944cb8943b8c7b5f6b84db"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "dd916787c89ac629e0b0e7e56bb079fc"
server_key_iv: "ec4de9437f00610b50522ff020d3d83d"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.663976 24529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:50.664703 24544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.664870 24529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:50.664963 24529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "eb7abd2b2e944cb8943b8c7b5f6b84db"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "dd916787c89ac629e0b0e7e56bb079fc"
server_key_iv: "ec4de9437f00610b50522ff020d3d83d"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.665043 24529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:50.677199 24529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:50.678928 24529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:50.679076 24529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:50.679289 24529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:50.679574 24529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:50.679625 24529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.679663 24529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:50.679688 24529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.685127 24529 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:33903
I20260925 14:53:50.685164 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:33903 every 8 connection(s)
I20260925 14:53:50.685530 24529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:50.693069 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24529
I20260925 14:53:50.693152 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:50.693362 21276 external_mini_cluster.cc:1428] Setting key f7bb4dade2b0ec03ca9acdcf419a53d6
I20260925 14:53:50.694370 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.686214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:44837 (local address 127.20.199.62:42353)
0925 14:53:50.686265 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:50.686268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:50.686467 (+ 199us) server_negotiation.cc:407] Connection header received
0925 14:53:50.686911 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:50.686912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:50.686922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:50.686970 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:50.687823 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.688322 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.689345 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.689616 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.689670 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.691663 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:50.691664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:50.691665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:50.691671 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:50.692848 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.693323 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.693324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.693324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.693384 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.693741 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.693742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.693742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.693873 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:50.693939 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:50.694208 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:53:50.694295 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:50.694777 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.686194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42353 (local address 127.20.199.2:44837)
0925 14:53:50.686354 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:50.686386 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:50.686833 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:50.687029 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:50.687032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:50.687566 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:50.687747 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:50.687751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.688397 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.688397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:50.689264 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:50.689269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.689684 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.689685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:50.689793 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.690303 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:50.690311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:50.691582 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:50.692916 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:50.692921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:50.692922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:50.693189 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:50.693510 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:50.693512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:50.693512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:50.693664 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:50.693935 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:50.693939 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:50.694053 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 14:53:50.694165 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:53:50.694373 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 14:53:50.695185 24658 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42353
I20260925 14:53:50.695289 24658 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:50.695464 24658 heartbeater.cc:511] Master 127.20.199.62:42353 requested a full tablet report, sending...
I20260925 14:53:50.695829 24332 ts_manager.cc:194] Registered new tserver with Master: eb7abd2b2e944cb8943b8c7b5f6b84db (127.20.199.2:33903)
I20260925 14:53:50.696182 24332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:44837
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:50.716754 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42353
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:50.802039 24663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:50.802264 24663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:50.802289 24663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:50.803530 24663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:50.803579 24663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:50.803635 24663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:50.805135 24663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:50.805454 24663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:50.805706 24663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:50.810725 24668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.810760 24671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:50.810761 24669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:50.811023 24663 server_base.cc:1034] running on GCE node
I20260925 14:53:50.811215 24663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:50.811447 24663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:50.812615 24663 hybrid_clock.cc:648] HybridClock initialized: now 1790348030812604 us; error 27 us; skew 500 ppm
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:50.815049 24663 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:50.815495 24663 webserver.cc:460] Webserver started at http://127.20.199.3:40789/ using document root <none> and password file <none>
I20260925 14:53:50.815680 24663 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:50.815718 24663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:50.815810 24663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:50.816808 24663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "a2a99e5d5bb04f85adbd771da32dd2dd"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "962aa71a3b525d6ad592dad57fe30c34"
server_key_iv: "c40ca33bbc3f03f433314f53403578f6"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.817118 24663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "a2a99e5d5bb04f85adbd771da32dd2dd"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "962aa71a3b525d6ad592dad57fe30c34"
server_key_iv: "c40ca33bbc3f03f433314f53403578f6"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.818612 24663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:50.819420 24678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.819604 24663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:50.819687 24663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "a2a99e5d5bb04f85adbd771da32dd2dd"
format_stamp: "Formatted at 2026-09-25 14:53:50 on dist-test-slave-8qt6"
server_key: "962aa71a3b525d6ad592dad57fe30c34"
server_key_iv: "c40ca33bbc3f03f433314f53403578f6"
server_key_version: "encryptionkey@0"
I20260925 14:53:50.819746 24663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:50.834903 24663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:50.837227 24663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:50.837363 24663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:50.837630 24663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:50.837975 24663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:50.838011 24663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.838034 24663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:50.838088 24663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:50.843837 24663 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:36141
I20260925 14:53:50.843918 24791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:36141 every 8 connection(s)
I20260925 14:53:50.844229 24663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:50.852223 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24663
I20260925 14:53:50.852336 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:50.852579 21276 external_mini_cluster.cc:1428] Setting key bc008d3011787740ffb8f0ff55c9261e
I20260925 14:53:50.853477 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.844966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:39647 (local address 127.20.199.62:42353)
0925 14:53:50.845023 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:50.845025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:50.845253 (+ 228us) server_negotiation.cc:407] Connection header received
0925 14:53:50.845893 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:50.845894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:50.845906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:50.845965 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:50.847008 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.847513 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.848526 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.848707 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.848762 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.850774 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:50.850776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:50.850776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:50.850782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:50.851927 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.852450 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.852451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.852451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.852504 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.852822 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.852823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.852824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.852941 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:50.853016 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:50.853270 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 14:53:50.853350 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:53:50.853669 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.844937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42353 (local address 127.20.199.3:39647)
0925 14:53:50.845118 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:50.845161 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:50.845765 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:50.846065 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:50.846069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:50.846679 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:50.846917 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:50.846924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.847590 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.847591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:50.848461 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:50.848463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.848775 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.848775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:50.848873 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.849327 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:50.849335 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:50.850685 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:50.852001 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:50.852023 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:50.852025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:50.852363 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:50.852585 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:50.852586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:50.852587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:50.852726 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:50.853006 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:50.853012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:50.853141 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 14:53:50.853247 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:53:50.853456 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260925 14:53:50.854146 24792 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42353
I20260925 14:53:50.854238 24792 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:50.854389 24792 heartbeater.cc:511] Master 127.20.199.62:42353 requested a full tablet report, sending...
I20260925 14:53:50.854712 24332 ts_manager.cc:194] Registered new tserver with Master: a2a99e5d5bb04f85adbd771da32dd2dd (127.20.199.3:36141)
I20260925 14:53:50.855042 24332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:39647
I20260925 14:53:50.864308 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348030, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:50.892928 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:50.880376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.20.199.62:42353)
0925 14:53:50.880436 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:50.880439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:50.880546 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:53:50.880598 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:50.880599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:50.880608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:50.880667 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:50.881206 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.881734 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.882427 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:50.882601 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:50.882649 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:50.890655 (+ 8006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:50.890656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:50.890657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:50.890664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:50.891849 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.892157 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.892157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.892158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.892204 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:50.892477 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:50.892477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:50.892477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:50.892621 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:50.892695 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:50.892799 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 14:53:50.892859 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 14:53:50.893193 24332 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59658
I20260925 14:53:50.893631 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24395
I20260925 14:53:50.897957 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24529
I20260925 14:53:50.902267 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24663
I20260925 14:53:50.906229 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24301
2026-09-25T14:53:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1829 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24804](info): set up 2 sockets
Sep 25 14:53:50 dist-test-slave-8qt6 krb5kdc[24804](info): commencing operation
krb5kdc: starting...
W20260925 14:53:51.942289 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:52.058444 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37185
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:37185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:52.138506 24817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:52.138751 24817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:52.138790 24817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:52.139925 24817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:52.139990 24817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:52.140017 24817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:52.140045 24817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:52.140067 24817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:52.141630 24817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:37185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:52.141897 24817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:52.142108 24817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:52.146837 24822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.146853 24823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.146836 24825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:52.146988 24817 server_base.cc:1034] running on GCE node
I20260925 14:53:52.147289 24817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:52.147555 24817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:52.148691 24817 hybrid_clock.cc:648] HybridClock initialized: now 1790348032148672 us; error 33 us; skew 500 ppm
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:52.151090 24817 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:52.151458 24817 webserver.cc:460] Webserver started at http://127.20.199.62:42683/ using document root <none> and password file <none>
I20260925 14:53:52.151643 24817 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:52.151684 24817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:52.151768 24817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:52.152760 24817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "86c4f9c422384003902def36ed195f58"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "4ad4730a91515961bdf7be6ad2c19e5f"
server_key_iv: "8fe1fe6b16b552c508419c3a212a786d"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.153047 24817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "86c4f9c422384003902def36ed195f58"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "4ad4730a91515961bdf7be6ad2c19e5f"
server_key_iv: "8fe1fe6b16b552c508419c3a212a786d"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.154320 24817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:52.154981 24832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.155150 24817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:52.155227 24817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "86c4f9c422384003902def36ed195f58"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "4ad4730a91515961bdf7be6ad2c19e5f"
server_key_iv: "8fe1fe6b16b552c508419c3a212a786d"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.155278 24817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:52.192638 24817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:52.194523 24817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:52.194639 24817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:52.198355 24817 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:37185
I20260925 14:53:52.198411 24884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:37185 every 8 connection(s)
I20260925 14:53:52.198876 24817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:52.199505 24885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:52.201819 24885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58: Bootstrap starting.
I20260925 14:53:52.202431 24885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:52.202716 24885 log.cc:824] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:52.203096 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24817
I20260925 14:53:52.203222 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:52.203409 21276 external_mini_cluster.cc:1428] Setting key 60fe5920bb7b734b97dd9440f8ebb475
I20260925 14:53:52.203464 24885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58: No bootstrap required, opened a new log
I20260925 14:53:52.205999 24885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c4f9c422384003902def36ed195f58" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37185 } }
I20260925 14:53:52.206135 24885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:52.206157 24885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c4f9c422384003902def36ed195f58, State: Initialized, Role: FOLLOWER
I20260925 14:53:52.206285 24885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c4f9c422384003902def36ed195f58" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37185 } }
I20260925 14:53:52.206357 24885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:52.206398 24885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:52.206445 24885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:52.206964 24885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c4f9c422384003902def36ed195f58" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37185 } }
I20260925 14:53:52.207079 24885 leader_election.cc:302] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86c4f9c422384003902def36ed195f58; no voters:
I20260925 14:53:52.207250 24885 leader_election.cc:288] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:52.207278 24890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:52.207401 24890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [term 1 LEADER]: Becoming Leader. State: Replica: 86c4f9c422384003902def36ed195f58, State: Running, Role: LEADER
I20260925 14:53:52.207480 24890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c4f9c422384003902def36ed195f58" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37185 } }
I20260925 14:53:52.207507 24885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:52.208698 24891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86c4f9c422384003902def36ed195f58. Latest consensus state: current_term: 1 leader_uuid: "86c4f9c422384003902def36ed195f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c4f9c422384003902def36ed195f58" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37185 } } }
I20260925 14:53:52.208801 24891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:52.208921 24892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86c4f9c422384003902def36ed195f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c4f9c422384003902def36ed195f58" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37185 } } }
I20260925 14:53:52.208999 24892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58 [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:52.209697 24906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:52.209774 24906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:52.209887 24905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:52.210234 24905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:52.211735 24905 catalog_manager.cc:1347] Generated new cluster ID: 6a42facdafcc4266a2844a7a78621056
I20260925 14:53:52.211787 24905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:52.215605 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.203892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.20.199.62:37185)
0925 14:53:52.204070 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:52.204079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:52.204114 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:52.204764 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:52.204772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:52.204825 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:52.204997 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:52.205687 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.206759 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.207738 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.208339 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.208482 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.211823 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:52.211827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:52.211827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:52.211839 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:52.213918 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.214307 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.214310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.214311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.214425 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.214619 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.214620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.214620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.214931 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:52.215085 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:52.215196 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:53:52.215330 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260925 14:53:52.239439 24905 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:52.240053 24905 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:52.242451 24905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86c4f9c422384003902def36ed195f58: Generated new TSK 0
I20260925 14:53:52.242643 24905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:52.276970 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37185
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:52.359081 24911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:52.359273 24911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:52.359295 24911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:52.360478 24911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:52.360522 24911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:52.360571 24911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:52.362179 24911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:52.362421 24911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:52.362644 24911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:52.367575 24916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:52.367667 24911 server_base.cc:1034] running on GCE node
W20260925 14:53:52.367723 24917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.367605 24919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:52.367966 24911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:52.368208 24911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:52.369387 24911 hybrid_clock.cc:648] HybridClock initialized: now 1790348032369373 us; error 40 us; skew 500 ppm
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:52.372045 24911 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:52.372525 24911 webserver.cc:460] Webserver started at http://127.20.199.1:37453/ using document root <none> and password file <none>
I20260925 14:53:52.372758 24911 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:52.372821 24911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:52.372969 24911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:52.374073 24911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "3e146fe022c84e0a85efe20e7bdf2a10"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "e04674ce70813c562ddb81996cab5761"
server_key_iv: "e8e765be1c5ac1471262a8f1256a0f6f"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.374429 24911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "3e146fe022c84e0a85efe20e7bdf2a10"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "e04674ce70813c562ddb81996cab5761"
server_key_iv: "e8e765be1c5ac1471262a8f1256a0f6f"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.375682 24911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:53:52.376362 24926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.376529 24911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.376626 24911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "3e146fe022c84e0a85efe20e7bdf2a10"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "e04674ce70813c562ddb81996cab5761"
server_key_iv: "e8e765be1c5ac1471262a8f1256a0f6f"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.376693 24911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:52.403556 24911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:52.405915 24911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:52.406075 24911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:52.406308 24911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:52.406620 24911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:52.406672 24911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.406713 24911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:52.406741 24911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.412671 24911 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:35307
I20260925 14:53:52.412714 25039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:35307 every 8 connection(s)
I20260925 14:53:52.413061 24911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:52.421798 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 24911
I20260925 14:53:52.421926 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:52.422063 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.413808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59727 (local address 127.20.199.62:37185)
0925 14:53:52.413870 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:52.413873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:52.414097 (+ 224us) server_negotiation.cc:407] Connection header received
0925 14:53:52.415005 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:52.415006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:52.415016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:52.415058 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:52.415630 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.416124 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.417061 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.417252 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.417300 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.419331 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:52.419332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:52.419333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:52.419340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:52.420685 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.421095 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.421096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.421097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.421152 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.421426 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.421427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.421427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.421543 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:52.421617 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:52.421882 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 14:53:52.421970 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:53:52.422143 21276 external_mini_cluster.cc:1428] Setting key ca6c5ee45aab167c07f1abb346817d4b
I20260925 14:53:52.422262 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.413802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37185 (local address 127.20.199.1:59727)
0925 14:53:52.413981 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:52.414019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:52.414903 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:52.415114 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:52.415117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:52.415372 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:52.415561 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:52.415565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.416196 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.416197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:52.416993 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:52.416995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.417318 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.417318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:52.417462 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.417933 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:52.417942 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:52.419233 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:52.420785 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:52.420791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:52.420793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:52.421022 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:52.421218 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:52.421219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:52.421219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:52.421327 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:52.421608 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:52.421614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:52.421755 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:53:52.421861 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:53:52.422022 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260925 14:53:52.422755 25040 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37185
I20260925 14:53:52.422878 25040 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:52.423081 25040 heartbeater.cc:511] Master 127.20.199.62:37185 requested a full tablet report, sending...
I20260925 14:53:52.423626 24849 ts_manager.cc:194] Registered new tserver with Master: 3e146fe022c84e0a85efe20e7bdf2a10 (127.20.199.1:35307)
I20260925 14:53:52.424228 24849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:59727
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:52.445667 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37185
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:52.524663 25045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:52.524869 25045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:52.524893 25045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:52.525985 25045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:52.526019 25045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:52.526046 25045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:52.527539 25045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:52.527717 25045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:52.527894 25045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:52.532680 25051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.532732 25053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.532685 25050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:52.532859 25045 server_base.cc:1034] running on GCE node
I20260925 14:53:52.533059 25045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:52.533258 25045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:52.534425 25045 hybrid_clock.cc:648] HybridClock initialized: now 1790348032534415 us; error 39 us; skew 500 ppm
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:52.536772 25045 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:52.537153 25045 webserver.cc:460] Webserver started at http://127.20.199.2:37671/ using document root <none> and password file <none>
I20260925 14:53:52.537349 25045 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:52.537438 25045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:52.537557 25045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:52.538553 25045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "d46da92cd69c4ae389413d4e38cb84d1"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "b0c52156a41d02268461822d6a4bf640"
server_key_iv: "ea6cee54ed231a4a70688f59d4f82b3e"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.538858 25045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "d46da92cd69c4ae389413d4e38cb84d1"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "b0c52156a41d02268461822d6a4bf640"
server_key_iv: "ea6cee54ed231a4a70688f59d4f82b3e"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.540145 25045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:53:52.540784 25060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.540943 25045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:52.541030 25045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "d46da92cd69c4ae389413d4e38cb84d1"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "b0c52156a41d02268461822d6a4bf640"
server_key_iv: "ea6cee54ed231a4a70688f59d4f82b3e"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.541098 25045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:52.579178 25045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:52.581383 25045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:52.581594 25045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:52.581845 25045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:52.582156 25045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:52.582207 25045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.582259 25045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:52.582293 25045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.587672 25045 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:41071
I20260925 14:53:52.587772 25173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:41071 every 8 connection(s)
I20260925 14:53:52.588063 25045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:52.590212 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25045
I20260925 14:53:52.590318 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:52.590512 21276 external_mini_cluster.cc:1428] Setting key 9aef0b7c8e37280cae4ba8074061dc6a
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:52.598168 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.588772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:51231 (local address 127.20.199.62:37185)
0925 14:53:52.588835 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:52.588837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:52.589016 (+ 179us) server_negotiation.cc:407] Connection header received
0925 14:53:52.589974 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:52.589975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:52.589985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:52.590040 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:52.590865 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.591506 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.592493 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.592694 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.592765 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.595297 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:52.595299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:52.595299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:52.595306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:52.596676 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.597157 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.597158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.597158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.597217 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.597530 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.597531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.597531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.597648 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:52.597702 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:52.598042 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 14:53:52.598103 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:52.598455 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.588747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37185 (local address 127.20.199.2:51231)
0925 14:53:52.588893 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:52.588934 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:52.589886 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:52.590126 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:52.590131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:52.590510 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:52.590773 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:52.590778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.591595 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.591596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:52.592382 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:52.592384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.592771 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.592772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:52.592909 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.593468 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:52.593477 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:52.595186 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:52.596761 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:52.596768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:52.596770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:52.597075 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:52.597284 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:52.597285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:52.597285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:52.597459 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:52.597716 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:52.597723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:52.597891 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 14:53:52.598007 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:53:52.598185 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:53:52.598968 25174 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37185
I20260925 14:53:52.599057 25174 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:52.599225 25174 heartbeater.cc:511] Master 127.20.199.62:37185 requested a full tablet report, sending...
I20260925 14:53:52.599591 24849 ts_manager.cc:194] Registered new tserver with Master: d46da92cd69c4ae389413d4e38cb84d1 (127.20.199.2:41071)
I20260925 14:53:52.599931 24849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:51231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:52.615442 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37185
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:52.695134 25179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:52.695325 25179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:52.695348 25179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:52.696476 25179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:52.696516 25179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:52.696564 25179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:52.698086 25179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:52.698328 25179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:52.698571 25179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:52.703403 25187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.703431 25184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:52.703406 25185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:52.703619 25179 server_base.cc:1034] running on GCE node
I20260925 14:53:52.703826 25179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:52.704058 25179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:52.705200 25179 hybrid_clock.cc:648] HybridClock initialized: now 1790348032705173 us; error 45 us; skew 500 ppm
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:52.707679 25179 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:52.708060 25179 webserver.cc:460] Webserver started at http://127.20.199.3:46743/ using document root <none> and password file <none>
I20260925 14:53:52.708321 25179 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:52.708415 25179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:52.708578 25179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:52.709791 25179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "e7dc0239963748bb97816f17f6861157"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "d23c2defd3cb35dcf7c86eb9dbc61635"
server_key_iv: "0ac56ae1157164275ca6f56faf7b8317"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.710137 25179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "e7dc0239963748bb97816f17f6861157"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "d23c2defd3cb35dcf7c86eb9dbc61635"
server_key_iv: "0ac56ae1157164275ca6f56faf7b8317"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.711422 25179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:52.712036 25194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.712199 25179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:52.712265 25179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "e7dc0239963748bb97816f17f6861157"
format_stamp: "Formatted at 2026-09-25 14:53:52 on dist-test-slave-8qt6"
server_key: "d23c2defd3cb35dcf7c86eb9dbc61635"
server_key_iv: "0ac56ae1157164275ca6f56faf7b8317"
server_key_version: "encryptionkey@0"
I20260925 14:53:52.712316 25179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:52.736044 25179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:52.738215 25179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:52.738340 25179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:52.738544 25179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:52.738884 25179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:52.738921 25179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.738968 25179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:52.738998 25179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:52.745528 25179 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:41631
I20260925 14:53:52.745569 25307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:41631 every 8 connection(s)
I20260925 14:53:52.745935 25179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:52.750159 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25179
I20260925 14:53:52.750252 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:52.750447 21276 external_mini_cluster.cc:1428] Setting key f81607c5f9e11ff6dde24493f1ec3c1f
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:52.754839 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.746668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:35281 (local address 127.20.199.62:37185)
0925 14:53:52.746725 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:52.746727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:52.746935 (+ 208us) server_negotiation.cc:407] Connection header received
0925 14:53:52.747852 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:52.747853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:52.747863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:52.747925 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:52.748525 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.749019 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.750035 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.750332 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.750413 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.752566 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:52.752568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:52.752568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:52.752574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:52.753765 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.754116 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.754117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.754117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.754164 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.754356 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.754357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.754357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.754456 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:52.754507 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:52.754717 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:53:52.754777 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:53:52.754998 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.746664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37185 (local address 127.20.199.3:35281)
0925 14:53:52.746810 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:52.746845 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:52.747765 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:52.747983 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:52.747986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:52.748246 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:52.748432 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:52.748436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.749097 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.749097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:52.749945 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:52.749947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.750429 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.750430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:52.750566 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.751113 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:52.751123 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:52.752491 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:52.753837 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:52.753842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:52.753844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:52.754058 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:52.754215 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:52.754216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:52.754216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:52.754305 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:52.754507 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:52.754510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:52.754617 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 14:53:52.754695 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:53:52.754805 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260925 14:53:52.755448 25308 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37185
I20260925 14:53:52.755542 25308 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:52.755725 25308 heartbeater.cc:511] Master 127.20.199.62:37185 requested a full tablet report, sending...
I20260925 14:53:52.756090 24849 ts_manager.cc:194] Registered new tserver with Master: e7dc0239963748bb97816f17f6861157 (127.20.199.3:41631)
I20260925 14:53:52.756479 24849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:35281
I20260925 14:53:52.762108 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:52.784574 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.777125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.20.199.1:35307)
0925 14:53:52.777285 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:52.777290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:52.777307 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:53:52.777332 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:52.777334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:52.777348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:52.777493 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:52.778014 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.778802 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.779534 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.779871 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.779923 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.782039 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:52.782048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:52.782048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:52.782060 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:52.783408 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.783752 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.783760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.783761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.783812 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.784017 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.784018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.784018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.784144 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:52.784293 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:52.784364 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:53:52.784452 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[24804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:52.792426 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:52.785784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.20.199.62:37185)
0925 14:53:52.785852 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:52.785854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:52.785862 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:52.785884 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:52.785885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:52.785893 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:52.785946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:52.786345 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.786824 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.787500 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:52.787685 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:52.787766 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:52.790236 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:52.790237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:52.790238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:52.790244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:52.791438 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.791756 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.791757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.791758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.791815 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:52.792002 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:52.792003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:52.792003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:52.792144 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:52.792220 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:52.792295 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:53:52.792361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 14:53:52.792666 24849 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60218
I20260925 14:53:52.793022 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24911
I20260925 14:53:52.797760 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25045
I20260925 14:53:52.802031 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25179
I20260925 14:53:52.806309 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 24817
2026-09-25T14:53:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1900 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[25320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[25320](info): set up 2 sockets
Sep 25 14:53:52 dist-test-slave-8qt6 krb5kdc[25320](info): commencing operation
krb5kdc: starting...
W20260925 14:53:53.841291 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:53 dist-test-slave-8qt6 krb5kdc[25320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:53.958364 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:43569
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:43569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:54.040320 25333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:54.040550 25333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:54.040586 25333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:54.041792 25333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:54.041854 25333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:54.041877 25333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:54.041901 25333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:54.041944 25333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:54.043443 25333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:43569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:43569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:54.043715 25333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:54.043944 25333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:54.048672 25338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.048691 25333 server_base.cc:1034] running on GCE node
W20260925 14:53:54.048703 25339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:54.048672 25341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.049098 25333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:54.049479 25333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:54.050648 25333 hybrid_clock.cc:648] HybridClock initialized: now 1790348034050629 us; error 30 us; skew 500 ppm
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:54.053236 25333 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:54.053637 25333 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:54.053709 25333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:54.053866 25333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:54.054932 25333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "9b2176a662d34010bd8ace3de03e6d87"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "312e41942081dce7eaa6c6a38fe0a77d"
server_key_iv: "9271b01c4fd783f8d856272eb9589682"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.055286 25333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "9b2176a662d34010bd8ace3de03e6d87"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "312e41942081dce7eaa6c6a38fe0a77d"
server_key_iv: "9271b01c4fd783f8d856272eb9589682"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.056582 25333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:54.057333 25347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.057566 25333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:54.057662 25333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "9b2176a662d34010bd8ace3de03e6d87"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "312e41942081dce7eaa6c6a38fe0a77d"
server_key_iv: "9271b01c4fd783f8d856272eb9589682"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.057742 25333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:54.074548 25333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:54.076987 25333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:54.077135 25333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:54.081094 25333 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:43569
I20260925 14:53:54.081120 25399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:43569 every 8 connection(s)
I20260925 14:53:54.081456 25333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:54.082336 25400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:54.083379 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25333
I20260925 14:53:54.083496 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:54.083680 21276 external_mini_cluster.cc:1428] Setting key 1b046bbe0aabf6cdc08cec89a5ca8d57
I20260925 14:53:54.084894 25400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87: Bootstrap starting.
I20260925 14:53:54.085672 25400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:54.085978 25400 log.cc:824] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:54.086671 25400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87: No bootstrap required, opened a new log
I20260925 14:53:54.088878 25400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2176a662d34010bd8ace3de03e6d87" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43569 } }
I20260925 14:53:54.089044 25400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:54.089082 25400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b2176a662d34010bd8ace3de03e6d87, State: Initialized, Role: FOLLOWER
I20260925 14:53:54.089217 25400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2176a662d34010bd8ace3de03e6d87" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43569 } }
I20260925 14:53:54.089303 25400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:54.089336 25400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:54.089387 25400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:54.090179 25400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2176a662d34010bd8ace3de03e6d87" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43569 } }
I20260925 14:53:54.090277 25400 leader_election.cc:302] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b2176a662d34010bd8ace3de03e6d87; no voters:
I20260925 14:53:54.090442 25400 leader_election.cc:288] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:54.090498 25405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:54.090659 25405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [term 1 LEADER]: Becoming Leader. State: Replica: 9b2176a662d34010bd8ace3de03e6d87, State: Running, Role: LEADER
I20260925 14:53:54.090711 25400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:54.090780 25405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2176a662d34010bd8ace3de03e6d87" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43569 } }
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:54.091132 25405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b2176a662d34010bd8ace3de03e6d87. Latest consensus state: current_term: 1 leader_uuid: "9b2176a662d34010bd8ace3de03e6d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2176a662d34010bd8ace3de03e6d87" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43569 } } }
I20260925 14:53:54.091145 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b2176a662d34010bd8ace3de03e6d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2176a662d34010bd8ace3de03e6d87" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43569 } } }
I20260925 14:53:54.091221 25405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:54.091251 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:54.092203 25412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:54.092670 25412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:54.094422 25412 catalog_manager.cc:1347] Generated new cluster ID: 669ab55ca3ef4e19bac162c440c420ce
I20260925 14:53:54.094483 25412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:54.095297 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.084219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.20.199.62:43569)
0925 14:53:54.084381 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.084394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.084422 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:53:54.084804 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.084810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.084855 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.085011 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:54.085859 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.086983 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.088053 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.088472 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.088565 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.091559 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:54.091563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:54.091563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:54.091574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:54.093668 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.094095 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.094098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.094099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.094193 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.094400 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.094401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.094402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.094696 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:54.094849 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.094958 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.095094 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 14:53:54.103309 25412 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:54.103977 25412 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:54.109357 25412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87: Generated new TSK 0
I20260925 14:53:54.109566 25412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:54.138648 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:43569
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:54.237084 25426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:54.237308 25426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:54.237334 25426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:54.238550 25426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:54.238596 25426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:54.238623 25426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:54.240154 25426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:54.240406 25426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:54.240648 25426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:54.246261 25432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:54.246263 25434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.246485 25426 server_base.cc:1034] running on GCE node
W20260925 14:53:54.246258 25431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.246788 25426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:54.247020 25426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:54.248207 25426 hybrid_clock.cc:648] HybridClock initialized: now 1790348034248180 us; error 56 us; skew 500 ppm
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:54.250819 25426 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:54.251157 25426 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:54.251223 25426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:54.251364 25426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:54.252372 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "20c50855aed94b0996d0276d6cb87cc2"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "2e6fee08f8c4ec3277fc6f7667a1cc04"
server_key_iv: "30ac13bd1b0e16f4f848bc652b5c3617"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.252722 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "20c50855aed94b0996d0276d6cb87cc2"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "2e6fee08f8c4ec3277fc6f7667a1cc04"
server_key_iv: "30ac13bd1b0e16f4f848bc652b5c3617"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.254122 25426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:54.254799 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.254956 25426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:54.255052 25426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "20c50855aed94b0996d0276d6cb87cc2"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "2e6fee08f8c4ec3277fc6f7667a1cc04"
server_key_iv: "30ac13bd1b0e16f4f848bc652b5c3617"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.255121 25426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:54.262703 25426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:54.264658 25426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:54.264806 25426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:54.265022 25426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:54.265308 25426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:54.265357 25426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.265419 25426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:54.265448 25426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.271040 25426 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:42719
I20260925 14:53:54.271076 25553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:42719 every 8 connection(s)
I20260925 14:53:54.271325 25426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:54.273906 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25426
I20260925 14:53:54.273991 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:54.274176 21276 external_mini_cluster.cc:1428] Setting key 0445c422d2eec6185dd6455c4d8be62e
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:54.280989 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.271955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:54565 (local address 127.20.199.62:43569)
0925 14:53:54.272025 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.272028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.272232 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:53:54.272680 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.272694 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.272704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.272747 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:54.273719 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.274363 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.275833 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.276071 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.276150 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.278311 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:54.278312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:54.278313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:54.278320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:54.279644 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.280048 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.280049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.280049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.280110 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.280355 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.280356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.280356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.280465 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:54.280544 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.280810 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.280902 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:53:54.281162 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.271944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:43569 (local address 127.20.199.1:54565)
0925 14:53:54.272122 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.272154 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.272591 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.272826 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.272829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.273068 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:54.273639 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.273644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.274472 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.274473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.275737 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.275740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.276159 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.276160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.276287 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.276800 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:54.276809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:54.278201 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:54.279746 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:54.279753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:54.279755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:54.279974 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:54.280172 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:54.280173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:54.280173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:54.280284 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:54.280536 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:54.280543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:54.280676 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.280787 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.280935 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260925 14:53:54.281744 25554 heartbeater.cc:347] Connected to a master server at 127.20.199.62:43569
I20260925 14:53:54.281845 25554 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:54.282006 25554 heartbeater.cc:511] Master 127.20.199.62:43569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
I20260925 14:53:54.282554 25364 ts_manager.cc:194] Registered new tserver with Master: 20c50855aed94b0996d0276d6cb87cc2 (127.20.199.1:42719)
I20260925 14:53:54.283128 25364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:54565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:54.300036 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:43569
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:54.380654 25559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:54.380857 25559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:54.380882 25559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:54.382032 25559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:54.382073 25559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:54.382099 25559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:54.383499 25559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:54.383665 25559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:54.383839 25559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:54.388578 25565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:54.388598 25564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.388644 25559 server_base.cc:1034] running on GCE node
W20260925 14:53:54.388584 25567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.389042 25559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:54.389276 25559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:54.390443 25559 hybrid_clock.cc:648] HybridClock initialized: now 1790348034390424 us; error 37 us; skew 500 ppm
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:54.393064 25559 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:54.393508 25559 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:54.393561 25559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:54.393718 25559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:54.394834 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "64a8e0d462f29452978edd4887623bcc"
server_key_iv: "c42cb33f7de623d003506fd5653830f6"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.395193 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "64a8e0d462f29452978edd4887623bcc"
server_key_iv: "c42cb33f7de623d003506fd5653830f6"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.396826 25559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:54.397558 25573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.397735 25559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:54.397837 25559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "64a8e0d462f29452978edd4887623bcc"
server_key_iv: "c42cb33f7de623d003506fd5653830f6"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.397918 25559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:54.425303 25559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:54.427672 25559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:54.427837 25559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:54.428076 25559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:54.428390 25559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:54.428440 25559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.428481 25559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:54.428508 25559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.434147 25559 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:34073
I20260925 14:53:54.434187 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:34073 every 8 connection(s)
I20260925 14:53:54.434468 25559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:54.434738 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25559
I20260925 14:53:54.434841 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:54.435050 21276 external_mini_cluster.cc:1428] Setting key 4e82cafe48d8be78bda4f762ad4811e6
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:53:54.445055 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.435217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:32787 (local address 127.20.199.62:43569)
0925 14:53:54.435263 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.435265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.435535 (+ 270us) server_negotiation.cc:407] Connection header received
0925 14:53:54.436178 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.436178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.436188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.436222 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:54.437552 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.438042 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.439581 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.439784 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.439844 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.442333 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:54.442334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:54.442335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:54.442341 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:54.443738 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.444117 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.444118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.444118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.444177 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.444418 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.444419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.444420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.444534 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:54.444602 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.444882 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.444962 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:54.445228 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.435181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:43569 (local address 127.20.199.2:32787)
0925 14:53:54.435380 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.435438 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.436084 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.436308 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.436312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.436711 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:54.437441 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.437448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.438130 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.438131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.439486 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.439490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.439872 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.439873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.440024 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.440503 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:54.440514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:54.442221 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:54.443818 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:54.443825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:54.443827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:54.444043 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:54.444241 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:54.444242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:54.444242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:54.444352 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:54.444605 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:54.444612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:54.444745 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.444851 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.445002 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260925 14:53:54.445744 25687 heartbeater.cc:347] Connected to a master server at 127.20.199.62:43569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
I20260925 14:53:54.445858 25687 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:54.446005 25687 heartbeater.cc:511] Master 127.20.199.62:43569 requested a full tablet report, sending...
I20260925 14:53:54.446359 25364 ts_manager.cc:194] Registered new tserver with Master: 7c7d7aa99d804054b0cc8dc05c59a0c3 (127.20.199.2:34073)
I20260925 14:53:54.446679 25364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:32787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:54.459501 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:43569
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:54.537544 25692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:54.537734 25692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:54.537756 25692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:54.538905 25692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:54.538949 25692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:54.539000 25692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:54.540446 25692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:54.540649 25692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:54.540863 25692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:54.545565 25698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:54.545580 25700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.545609 25692 server_base.cc:1034] running on GCE node
W20260925 14:53:54.545583 25697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.545984 25692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:54.546197 25692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:54.547348 25692 hybrid_clock.cc:648] HybridClock initialized: now 1790348034547344 us; error 35 us; skew 500 ppm
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:54.549806 25692 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:54.550164 25692 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:54.550230 25692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:54.550380 25692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:54.551390 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "b78ab777915e40e5bbe53dcb4d59fa27"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "bec2ac8674861333d91eaa61d6464a66"
server_key_iv: "ed72394a10dbf305c023b29b4d40e5d8"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.551733 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "b78ab777915e40e5bbe53dcb4d59fa27"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "bec2ac8674861333d91eaa61d6464a66"
server_key_iv: "ed72394a10dbf305c023b29b4d40e5d8"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.552951 25692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:53:54.553710 25706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.553865 25692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:54.553954 25692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "b78ab777915e40e5bbe53dcb4d59fa27"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "bec2ac8674861333d91eaa61d6464a66"
server_key_iv: "ed72394a10dbf305c023b29b4d40e5d8"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.554028 25692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:54.570541 25692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:54.572309 25692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:54.572465 25692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:54.572685 25692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:54.572984 25692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:54.573035 25692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.573074 25692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:54.573101 25692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.578986 25692 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:41989
I20260925 14:53:54.579025 25819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:41989 every 8 connection(s)
I20260925 14:53:54.579276 25692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:54.584007 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25692
I20260925 14:53:54.584129 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:54.584343 21276 external_mini_cluster.cc:1428] Setting key 94e886ac5eac3919f334804bfc6c604c
Sep 25 14:53:54 dist-test-slave-8qt6 krb5kdc[25320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348034, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:54.588471 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.579923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:55679 (local address 127.20.199.62:43569)
0925 14:53:54.579986 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.579988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.580189 (+ 201us) server_negotiation.cc:407] Connection header received
0925 14:53:54.580699 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.580701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.580713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.580769 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:54.581776 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.582267 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.583262 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.583474 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.583557 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.585908 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:54.585909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:54.585910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:54.585917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:54.587403 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.587730 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.587730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.587731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.587779 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.587972 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.587972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.587973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.588077 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:54.588126 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.588348 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.588409 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:54.588639 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.579912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:43569 (local address 127.20.199.3:55679)
0925 14:53:54.580074 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.580107 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.580595 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.580850 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.580854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.581116 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:54.581681 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.581687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.582366 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.582367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.583168 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.583170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.583566 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.583567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.583691 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.584233 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:54.584244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:54.585808 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:54.587469 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:54.587473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:54.587475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:54.587672 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:54.587835 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:54.587836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:54.587836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:54.587923 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:54.588127 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:54.588131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:54.588240 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.588320 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.588445 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260925 14:53:54.589056 25820 heartbeater.cc:347] Connected to a master server at 127.20.199.62:43569
I20260925 14:53:54.589151 25820 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:54.589303 25820 heartbeater.cc:511] Master 127.20.199.62:43569 requested a full tablet report, sending...
I20260925 14:53:54.589660 25364 ts_manager.cc:194] Registered new tserver with Master: b78ab777915e40e5bbe53dcb4d59fa27 (127.20.199.3:41989)
I20260925 14:53:54.590026 25364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:55679
I20260925 14:53:54.596575 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:54.603385 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.597787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.20.199.62:43569)
0925 14:53:54.597848 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.597851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.597913 (+ 62us) server_negotiation.cc:407] Connection header received
0925 14:53:54.597952 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.597953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.597960 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.598009 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:54.598503 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.598953 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.599611 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.599781 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.599828 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.601000 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:54.601001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:54.601001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:54.601006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:54.602363 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.602672 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.602673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.602674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.602727 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:54.602906 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:54.602907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:54.602907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:54.603045 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:54.603119 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.603199 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.603276 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:53:54.604955 25364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:54.606539 25364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:54.613037 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.608727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:41989 (local address 127.0.0.1:43786)
0925 14:53:54.608908 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.608917 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.608954 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.609240 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.609252 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.609254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:54.609336 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.609337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.610676 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.610676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.611993 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.611996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.612811 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.612871 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.612874 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.612919 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.612953 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":50,"thread_start_us":81,"threads_started":1}
I20260925 14:53:54.613124 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.608643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.20.199.1:42719)
0925 14:53:54.609137 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.609141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.609149 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:54.609179 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.609181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.609213 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.609292 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:54.609855 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.610533 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612113 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.612595 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612682 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.612743 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.612988 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.613033 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":87,"threads_started":1}
I20260925 14:53:54.613125 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.608729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.20.199.2:34073)
0925 14:53:54.609457 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.609461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.609469 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:54.609523 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.609525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.609541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.609618 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:54.609749 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.610786 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.611930 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.612655 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612913 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.612970 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.612996 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.613034 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":127,"threads_started":1}
I20260925 14:53:54.613353 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.608935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.20.199.3:41989)
0925 14:53:54.609093 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.609097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.609105 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:54.609135 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.609137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.609149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.609244 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:54.609494 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.610592 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612077 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.612748 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612814 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.612872 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.613238 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.613290 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260925 14:53:54.613674 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.608546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:42719 (local address 127.0.0.1:45026)
0925 14:53:54.608705 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.608744 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.608804 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.609714 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.609715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.609717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:54.609773 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.609774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.610628 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.610629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.612030 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.612032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.612681 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.612774 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.612789 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.612856 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.612930 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":29,"thread_start_us":58,"threads_started":1}
I20260925 14:53:54.614037 25754 tablet_service.cc:1467] Processing CreateTablet for tablet 8c30a10b07604630a30b1d7976380e11 (DEFAULT_TABLE table=test-table [id=f1b39827b8de414996969c69c270502d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:54.614266 25488 tablet_service.cc:1467] Processing CreateTablet for tablet 8c30a10b07604630a30b1d7976380e11 (DEFAULT_TABLE table=test-table [id=f1b39827b8de414996969c69c270502d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:54.614372 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c30a10b07604630a30b1d7976380e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:54.614549 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c30a10b07604630a30b1d7976380e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:54.616902 25834 tablet_bootstrap.cc:492] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Bootstrap starting.
I20260925 14:53:54.616940 25835 tablet_bootstrap.cc:492] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2: Bootstrap starting.
I20260925 14:53:54.613037 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.608546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:34073 (local address 127.0.0.1:47422)
0925 14:53:54.608859 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.608871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.608924 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.609615 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.609616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.609618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:54.609656 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.609657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.610893 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.610895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.611848 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.611850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.612759 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.612760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.612831 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.612835 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.612866 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.612942 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":33,"thread_start_us":55,"threads_started":1}
I20260925 14:53:54.617640 25835 tablet_bootstrap.cc:654] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:54.617769 25834 tablet_bootstrap.cc:654] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:54.617934 25621 tablet_service.cc:1467] Processing CreateTablet for tablet 8c30a10b07604630a30b1d7976380e11 (DEFAULT_TABLE table=test-table [id=f1b39827b8de414996969c69c270502d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:54.618219 25835 log.cc:824] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:54.618321 25621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c30a10b07604630a30b1d7976380e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:54.618376 25834 log.cc:824] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:54.619068 25834 tablet_bootstrap.cc:492] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: No bootstrap required, opened a new log
I20260925 14:53:54.619141 25834 ts_tablet_manager.cc:1397] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:53:54.621109 25834 raft_consensus.cc:348] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.621296 25834 raft_consensus.cc:374] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:54.621333 25834 raft_consensus.cc:729] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78ab777915e40e5bbe53dcb4d59fa27, State: Initialized, Role: FOLLOWER
I20260925 14:53:54.621481 25834 consensus_queue.cc:260] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.621759 25834 ts_tablet_manager.cc:1428] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:53:54.621888 25820 heartbeater.cc:503] Master 127.20.199.62:43569 was elected leader, sending a full tablet report...
I20260925 14:53:54.621985 25838 tablet_bootstrap.cc:492] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Bootstrap starting.
I20260925 14:53:54.622211 25835 tablet_bootstrap.cc:492] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2: No bootstrap required, opened a new log
I20260925 14:53:54.622285 25835 ts_tablet_manager.cc:1397] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:53:54.622568 25838 tablet_bootstrap.cc:654] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:54.623041 25838 log.cc:824] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:54.623729 25838 tablet_bootstrap.cc:492] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: No bootstrap required, opened a new log
I20260925 14:53:54.623806 25838 ts_tablet_manager.cc:1397] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:54.624179 25835 raft_consensus.cc:348] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.624303 25835 raft_consensus.cc:374] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:54.624325 25835 raft_consensus.cc:729] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c50855aed94b0996d0276d6cb87cc2, State: Initialized, Role: FOLLOWER
I20260925 14:53:54.624450 25835 consensus_queue.cc:260] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.624719 25835 ts_tablet_manager.cc:1428] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:53:54.624801 25554 heartbeater.cc:503] Master 127.20.199.62:43569 was elected leader, sending a full tablet report...
I20260925 14:53:54.625582 25838 raft_consensus.cc:348] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.625739 25838 raft_consensus.cc:374] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:54.625771 25838 raft_consensus.cc:729] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7d7aa99d804054b0cc8dc05c59a0c3, State: Initialized, Role: FOLLOWER
I20260925 14:53:54.625859 25838 consensus_queue.cc:260] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.626080 25838 ts_tablet_manager.cc:1428] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:54.626183 25687 heartbeater.cc:503] Master 127.20.199.62:43569 was elected leader, sending a full tablet report...
W20260925 14:53:54.684964 25688 tablet.cc:2369] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:54.699184 25842 raft_consensus.cc:482] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:54.699308 25842 raft_consensus.cc:504] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.699604 25842 leader_election.cc:288] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20c50855aed94b0996d0276d6cb87cc2 (127.20.199.1:42719), b78ab777915e40e5bbe53dcb4d59fa27 (127.20.199.3:41989)
I20260925 14:53:54.702737 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.699906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57249 (local address 127.20.199.1:42719)
0925 14:53:54.699977 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.699980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.699990 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:54.700021 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.700022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.700033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.700093 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:54.700340 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.700952 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.702030 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.702393 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.702474 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.702482 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.702625 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.702665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:53:54.702747 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.700112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:58537 (local address 127.20.199.3:41989)
0925 14:53:54.700153 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.700154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.700274 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:53:54.700319 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.700320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.700331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.700379 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:54.700513 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.700980 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.702016 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.702391 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.702469 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.702478 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.702649 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.702688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:54.702744 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.699832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:42719 (local address 127.20.199.2:57249)
0925 14:53:54.699887 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.699902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.699968 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.700165 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.700166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.700168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:54.700233 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.700235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.701022 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.701023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.701930 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.701933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.702472 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.702473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.702556 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.702559 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.702601 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.702654 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":38}
I20260925 14:53:54.702757 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.699955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:41989 (local address 127.20.199.2:58537)
0925 14:53:54.700204 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.700211 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:54.700255 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:54.700382 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:54.700383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:54.700385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:54.700450 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.700452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.701080 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.701081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.701931 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:54.701933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.702481 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.702482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:54.702566 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.702569 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:54.702635 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:53:54.702671 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260925 14:53:54.703184 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c30a10b07604630a30b1d7976380e11" candidate_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" is_pre_election: true
I20260925 14:53:54.703346 25774 raft_consensus.cc:2454] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7d7aa99d804054b0cc8dc05c59a0c3 in term 0.
I20260925 14:53:54.703593 25577 leader_election.cc:302] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7d7aa99d804054b0cc8dc05c59a0c3, b78ab777915e40e5bbe53dcb4d59fa27; no voters:
I20260925 14:53:54.703577 25508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c30a10b07604630a30b1d7976380e11" candidate_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c50855aed94b0996d0276d6cb87cc2" is_pre_election: true
I20260925 14:53:54.703714 25508 raft_consensus.cc:2454] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7d7aa99d804054b0cc8dc05c59a0c3 in term 0.
I20260925 14:53:54.703725 25842 raft_consensus.cc:2781] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:54.703791 25842 raft_consensus.cc:482] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:54.703835 25842 raft_consensus.cc:3037] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:54.704486 25842 raft_consensus.cc:504] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.704620 25842 leader_election.cc:288] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [CANDIDATE]: Term 1 election: Requested vote from peers 20c50855aed94b0996d0276d6cb87cc2 (127.20.199.1:42719), b78ab777915e40e5bbe53dcb4d59fa27 (127.20.199.3:41989)
I20260925 14:53:54.704792 25508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c30a10b07604630a30b1d7976380e11" candidate_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c50855aed94b0996d0276d6cb87cc2"
I20260925 14:53:54.704802 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c30a10b07604630a30b1d7976380e11" candidate_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b78ab777915e40e5bbe53dcb4d59fa27"
I20260925 14:53:54.704862 25508 raft_consensus.cc:3037] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:54.704895 25774 raft_consensus.cc:3037] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:54.705600 25774 raft_consensus.cc:2454] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7d7aa99d804054b0cc8dc05c59a0c3 in term 1.
I20260925 14:53:54.705643 25508 raft_consensus.cc:2454] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7d7aa99d804054b0cc8dc05c59a0c3 in term 1.
I20260925 14:53:54.705768 25577 leader_election.cc:302] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c50855aed94b0996d0276d6cb87cc2, 7c7d7aa99d804054b0cc8dc05c59a0c3; no voters:
I20260925 14:53:54.705871 25842 raft_consensus.cc:2781] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:54.706049 25842 raft_consensus.cc:686] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7d7aa99d804054b0cc8dc05c59a0c3, State: Running, Role: LEADER
I20260925 14:53:54.706162 25842 consensus_queue.cc:237] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } }
I20260925 14:53:54.706948 25364 catalog_manager.cc:5495] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c7d7aa99d804054b0cc8dc05c59a0c3 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c7d7aa99d804054b0cc8dc05c59a0c3" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:54.730724 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.728294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.20.199.2:34073)
0925 14:53:54.728368 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.728369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.728378 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:54.728412 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.728412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.728429 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.728504 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:54.728633 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.729104 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.729917 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:54.730108 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:54.730186 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:54.730355 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:54.730454 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:54.730528 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.730631 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.730656 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:53:54.732498 25508 raft_consensus.cc:1261] T 8c30a10b07604630a30b1d7976380e11 P 20c50855aed94b0996d0276d6cb87cc2 [term 1 FOLLOWER]: Refusing update from remote peer 7c7d7aa99d804054b0cc8dc05c59a0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:54.732498 25774 raft_consensus.cc:1261] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 1 FOLLOWER]: Refusing update from remote peer 7c7d7aa99d804054b0cc8dc05c59a0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:54.732872 25842 consensus_queue.cc:1035] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b78ab777915e40e5bbe53dcb4d59fa27" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:54.733042 25844 consensus_queue.cc:1035] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c50855aed94b0996d0276d6cb87cc2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:54.736117 25848 mvcc.cc:205] Tried to move back new op lower bound from 7333265550261506048 to 7333265550157115392. Current Snapshot: MvccSnapshot[applied={T|T < 7333265550261506048}]
I20260925 14:53:54.736235 25849 mvcc.cc:205] Tried to move back new op lower bound from 7333265550261506048 to 7333265550157115392. Current Snapshot: MvccSnapshot[applied={T|T < 7333265550261506048}]
I20260925 14:53:54.737558 25363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:54.737637 25363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:54.738386 25363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b2176a662d34010bd8ace3de03e6d87: Sending DeleteTablet for 3 replicas of tablet 8c30a10b07604630a30b1d7976380e11
I20260925 14:53:54.738791 25488 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c30a10b07604630a30b1d7976380e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:54 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:45026
I20260925 14:53:54.738795 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c30a10b07604630a30b1d7976380e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:54 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:43786
I20260925 14:53:54.738951 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25426
I20260925 14:53:54.739037 25621 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c30a10b07604630a30b1d7976380e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:54 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:47422
I20260925 14:53:54.739756 25859 tablet_replica.cc:330] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: stopping tablet replica
I20260925 14:53:54.739849 25859 raft_consensus.cc:2229] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:54.739959 25858 tablet_replica.cc:330] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: stopping tablet replica
I20260925 14:53:54.739966 25859 raft_consensus.cc:2258] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:54.740094 25858 raft_consensus.cc:2229] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:54.740231 25858 raft_consensus.cc:2258] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:54.740417 25859 ts_tablet_manager.cc:1905] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:54.740721 25858 ts_tablet_manager.cc:1905] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:54.742055 25859 ts_tablet_manager.cc:1918] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:54.742122 25859 log.cc:1196] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/8c30a10b07604630a30b1d7976380e11
I20260925 14:53:54.742393 25859 ts_tablet_manager.cc:1939] T 8c30a10b07604630a30b1d7976380e11 P 7c7d7aa99d804054b0cc8dc05c59a0c3: Deleting consensus metadata
I20260925 14:53:54.742769 25348 catalog_manager.cc:4841] TS 7c7d7aa99d804054b0cc8dc05c59a0c3 (127.20.199.2:34073): tablet 8c30a10b07604630a30b1d7976380e11 (table test-table [id=f1b39827b8de414996969c69c270502d]) successfully deleted
I20260925 14:53:54.742854 25858 ts_tablet_manager.cc:1918] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:54.742926 25858 log.cc:1196] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/8c30a10b07604630a30b1d7976380e11
I20260925 14:53:54.743158 25858 ts_tablet_manager.cc:1939] T 8c30a10b07604630a30b1d7976380e11 P b78ab777915e40e5bbe53dcb4d59fa27: Deleting consensus metadata
I20260925 14:53:54.743784 25351 catalog_manager.cc:4841] TS b78ab777915e40e5bbe53dcb4d59fa27 (127.20.199.3:41989): tablet 8c30a10b07604630a30b1d7976380e11 (table test-table [id=f1b39827b8de414996969c69c270502d]) successfully deleted
W20260925 14:53:54.748327 25351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:42719 (error 108)
I20260925 14:53:54.748525 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25559
I20260925 14:53:54.749161 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.748667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:42719 (local address 127.0.0.1:45040)
0925 14:53:54.749080 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:54.749121 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260925 14:53:54.749462 25351 catalog_manager.cc:4584] TS 20c50855aed94b0996d0276d6cb87cc2 (127.20.199.1:42719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c30a10b07604630a30b1d7976380e11: Network error: Client connection negotiation failed: client connection to 127.20.199.1:42719: connect: Connection refused (error 111)
I20260925 14:53:54.752681 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25692
I20260925 14:53:54.757057 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25333
2026-09-25T14:53:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1951 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:54Z Disabled control of system clock
I20260925 14:53:54.784080 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34111
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:53:54.865311 25864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:54.865586 25864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:54.865609 25864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:54.866752 25864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:54.866793 25864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:54.866806 25864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:54.866848 25864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:54.866873 25864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:54.868214 25864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:54.868465 25864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:54.868757 25864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:54.873457 25869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:54.873468 25872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:54.873468 25870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:54.873649 25864 server_base.cc:1034] running on GCE node
I20260925 14:53:54.873869 25864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:54.874148 25864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:54.875335 25864 hybrid_clock.cc:648] HybridClock initialized: now 1790348034875328 us; error 38 us; skew 500 ppm
I20260925 14:53:54.876724 25864 webserver.cc:460] Webserver started at http://127.20.199.62:40255/ using document root <none> and password file <none>
I20260925 14:53:54.877034 25864 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:54.877110 25864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:54.877271 25864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:54.878508 25864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "c83003e4486944e0bc643f82a5066519"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "10b7bdddb4eb0b266a8fdd7707604843"
server_key_iv: "4feb8924cb9bcf452888105f63cb9dfd"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.878901 25864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "c83003e4486944e0bc643f82a5066519"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "10b7bdddb4eb0b266a8fdd7707604843"
server_key_iv: "4feb8924cb9bcf452888105f63cb9dfd"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.880298 25864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:54.881038 25878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:54.881230 25864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:54.881321 25864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "c83003e4486944e0bc643f82a5066519"
format_stamp: "Formatted at 2026-09-25 14:53:54 on dist-test-slave-8qt6"
server_key: "10b7bdddb4eb0b266a8fdd7707604843"
server_key_iv: "4feb8924cb9bcf452888105f63cb9dfd"
server_key_version: "encryptionkey@0"
I20260925 14:53:54.881398 25864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:54.884617 25864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:54.884881 25864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:54.885005 25864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:54.888796 25864 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34111
I20260925 14:53:54.888828 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34111 every 8 connection(s)
I20260925 14:53:54.889127 25864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:54.889756 25931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:54.892356 25931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519: Bootstrap starting.
I20260925 14:53:54.892896 25931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:54.893172 25931 log.cc:824] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:54.893875 25931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519: No bootstrap required, opened a new log
I20260925 14:53:54.895383 25931 raft_consensus.cc:348] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83003e4486944e0bc643f82a5066519" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34111 } }
I20260925 14:53:54.895510 25931 raft_consensus.cc:374] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:54.895531 25931 raft_consensus.cc:729] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83003e4486944e0bc643f82a5066519, State: Initialized, Role: FOLLOWER
I20260925 14:53:54.895654 25931 consensus_queue.cc:260] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83003e4486944e0bc643f82a5066519" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34111 } }
I20260925 14:53:54.895727 25931 raft_consensus.cc:388] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:54.895749 25931 raft_consensus.cc:482] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:54.895781 25931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:54.896245 25931 raft_consensus.cc:504] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83003e4486944e0bc643f82a5066519" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34111 } }
I20260925 14:53:54.896332 25931 leader_election.cc:302] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c83003e4486944e0bc643f82a5066519; no voters:
I20260925 14:53:54.896449 25931 leader_election.cc:288] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:54.896497 25934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:54.896661 25934 raft_consensus.cc:686] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [term 1 LEADER]: Becoming Leader. State: Replica: c83003e4486944e0bc643f82a5066519, State: Running, Role: LEADER
I20260925 14:53:54.896739 25931 sys_catalog.cc:564] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:54.896740 25934 consensus_queue.cc:237] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83003e4486944e0bc643f82a5066519" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34111 } }
I20260925 14:53:54.897097 25936 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c83003e4486944e0bc643f82a5066519. Latest consensus state: current_term: 1 leader_uuid: "c83003e4486944e0bc643f82a5066519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83003e4486944e0bc643f82a5066519" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34111 } } }
I20260925 14:53:54.897183 25936 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:54.897329 25935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c83003e4486944e0bc643f82a5066519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83003e4486944e0bc643f82a5066519" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34111 } } }
I20260925 14:53:54.897415 25935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:54.897419 25940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:54.897907 25940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:54.898108 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25864
I20260925 14:53:54.898213 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:54.898401 21276 external_mini_cluster.cc:1428] Setting key 3a9d97f79ec1210c40a5f75d2d4a6269
I20260925 14:53:54.899488 25940 catalog_manager.cc:1347] Generated new cluster ID: d34d81c7591449368264fe476e6ed5b3
I20260925 14:53:54.899554 25940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:54.902956 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:54.898984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.20.199.62:34111)
0925 14:53:54.899296 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:54.899304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:54.899327 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:53:54.899437 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:54.901469 (+ 2032us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:54.901530 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:54.901682 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:54.902418 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:54.902421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:54.902421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:54.902425 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:54.902462 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:54.902476 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:54.902547 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:54.902708 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:53:54.902768 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":59,"threads_started":1}
I20260925 14:53:54.917908 25940 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:54.918082 25940 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:54.925976 25940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519: Generated new TSK 0
I20260925 14:53:54.926172 25940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:54.933516 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34111
--builtin_ntp_servers=127.20.199.20:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:53:55.028839 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:55.029050 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:55.029073 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:55.030320 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:55.030364 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:55.030390 25955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:55.032064 25955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:55.032341 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:55.032670 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:55.038936 25960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.038990 25961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.039201 25963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.039402 25955 server_base.cc:1034] running on GCE node
I20260925 14:53:55.039669 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:55.039938 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:55.041111 25955 hybrid_clock.cc:648] HybridClock initialized: now 1790348035041096 us; error 35 us; skew 500 ppm
I20260925 14:53:55.042680 25955 webserver.cc:460] Webserver started at http://127.20.199.1:45577/ using document root <none> and password file <none>
I20260925 14:53:55.042924 25955 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:55.042990 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:55.043131 25955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:55.044246 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "92818dabf43c436d89bb78379b772dc7"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "70ee963cf1a877a4dc45551f760d99c7"
server_key_iv: "ff706ed4a6c66d14920fd80b4699a0bc"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.044623 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "92818dabf43c436d89bb78379b772dc7"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "70ee963cf1a877a4dc45551f760d99c7"
server_key_iv: "ff706ed4a6c66d14920fd80b4699a0bc"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.046056 25955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:55.046841 25969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.047081 25955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:55.047183 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "92818dabf43c436d89bb78379b772dc7"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "70ee963cf1a877a4dc45551f760d99c7"
server_key_iv: "ff706ed4a6c66d14920fd80b4699a0bc"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.047255 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:55.051113 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:55.051353 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:55.051451 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:55.051601 25955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:55.051847 25955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:55.051877 25955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.051898 25955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:55.051911 25955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.057432 25955 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:32919
I20260925 14:53:55.057457 26082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:32919 every 8 connection(s)
I20260925 14:53:55.057821 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:55.059003 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 25955
I20260925 14:53:55.059118 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:55.059327 21276 external_mini_cluster.cc:1428] Setting key 5ac4bc16db825d8ef66f7f355c27b3ed
I20260925 14:53:55.060582 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34111
--builtin_ntp_servers=127.20.199.20:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 14:53:55.061460 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.058543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:56811 (local address 127.20.199.62:34111)
0925 14:53:55.058595 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.058597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.059493 (+ 896us) server_negotiation.cc:407] Connection header received
0925 14:53:55.059563 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.059564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.059572 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.059628 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.061106 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.061107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.061108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.061112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.061152 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.061157 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.061268 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.061350 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.061366 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:53:55.061682 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.058534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34111 (local address 127.20.199.1:56811)
0925 14:53:55.058724 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.058756 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.059452 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.059722 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.059726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.059733 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.060899 (+ 1166us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.060900 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.060941 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.060977 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.061271 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.061276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.061280 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.061357 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.061443 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260925 14:53:55.062603 26083 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34111
I20260925 14:53:55.062711 26083 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:55.062852 26083 heartbeater.cc:511] Master 127.20.199.62:34111 requested a full tablet report, sending...
I20260925 14:53:55.063380 25895 ts_manager.cc:194] Registered new tserver with Master: 92818dabf43c436d89bb78379b772dc7 (127.20.199.1:32919)
W20260925 14:53:55.140776 26086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:55.140996 26086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:55.141019 26086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:55.142246 26086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:55.142292 26086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:55.142340 26086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:55.143682 26086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:55.143910 26086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:55.144194 26086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:55.148977 26094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.149024 26086 server_base.cc:1034] running on GCE node
W20260925 14:53:55.149111 26091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.149006 26092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.149338 26086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:55.149619 26086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:55.150768 26086 hybrid_clock.cc:648] HybridClock initialized: now 1790348035150767 us; error 36 us; skew 500 ppm
I20260925 14:53:55.152031 26086 webserver.cc:460] Webserver started at http://127.20.199.2:40491/ using document root <none> and password file <none>
I20260925 14:53:55.152262 26086 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:55.152347 26086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:55.152480 26086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:55.153584 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "badf65d103d14146a34fd1e9f6afde7b"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "4a6b3528b723074a4f71ae880902b62c"
server_key_iv: "bd153c255acbf96a2f112a0810f62f1b"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.153937 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "badf65d103d14146a34fd1e9f6afde7b"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "4a6b3528b723074a4f71ae880902b62c"
server_key_iv: "bd153c255acbf96a2f112a0810f62f1b"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.155242 26086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:55.155972 26100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.156147 26086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:55.156237 26086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "badf65d103d14146a34fd1e9f6afde7b"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "4a6b3528b723074a4f71ae880902b62c"
server_key_iv: "bd153c255acbf96a2f112a0810f62f1b"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.156334 26086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:55.159760 26086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:55.160018 26086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:55.160154 26086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:55.160357 26086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:55.160642 26086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:55.160694 26086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.160730 26086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:55.160756 26086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.166563 26086 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:33869
I20260925 14:53:55.166605 26213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:33869 every 8 connection(s)
I20260925 14:53:55.166898 26086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:55.169764 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.167548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:34885 (local address 127.20.199.62:34111)
0925 14:53:55.167623 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.167625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.167814 (+ 189us) server_negotiation.cc:407] Connection header received
0925 14:53:55.168259 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.168261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.168266 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.168335 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.169565 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.169566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.169566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.169569 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.169582 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.169586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.169632 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.169703 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.169714 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:53:55.169983 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.167523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34111 (local address 127.20.199.2:34885)
0925 14:53:55.167702 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.167732 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.168181 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.168386 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.168389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.168396 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.169387 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.169388 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.169461 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.169496 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.169649 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.169651 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.169653 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.169712 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.169757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260925 14:53:55.170480 26214 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34111
I20260925 14:53:55.170572 26214 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:55.170708 26214 heartbeater.cc:511] Master 127.20.199.62:34111 requested a full tablet report, sending...
I20260925 14:53:55.171056 25895 ts_manager.cc:194] Registered new tserver with Master: badf65d103d14146a34fd1e9f6afde7b (127.20.199.2:33869)
I20260925 14:53:55.175575 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26086
I20260925 14:53:55.175676 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:55.175856 21276 external_mini_cluster.cc:1428] Setting key 60411f029d092d60655b84a223289c06
I20260925 14:53:55.176740 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34111
--builtin_ntp_servers=127.20.199.20:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:53:55.255445 26217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:55.255636 26217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:55.255658 26217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:55.256794 26217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:55.256841 26217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:55.256889 26217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:55.258219 26217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:55.258474 26217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:55.258781 26217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:55.263926 26225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.263918 26223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.263938 26217 server_base.cc:1034] running on GCE node
W20260925 14:53:55.263918 26222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.264348 26217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:55.264602 26217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:55.265756 26217 hybrid_clock.cc:648] HybridClock initialized: now 1790348035265745 us; error 28 us; skew 500 ppm
I20260925 14:53:55.267024 26217 webserver.cc:460] Webserver started at http://127.20.199.3:32809/ using document root <none> and password file <none>
I20260925 14:53:55.267277 26217 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:55.267325 26217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:55.267459 26217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:55.268513 26217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "09aa9b2879144383ab9eb07efd1322c6"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "4bbb0099cc26bfafdc173eb2fe3c7c80"
server_key_iv: "453ed0a15eaef767026604c376aba281"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.268874 26217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "09aa9b2879144383ab9eb07efd1322c6"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "4bbb0099cc26bfafdc173eb2fe3c7c80"
server_key_iv: "453ed0a15eaef767026604c376aba281"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.270262 26217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:55.270936 26231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.271112 26217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:55.271183 26217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "09aa9b2879144383ab9eb07efd1322c6"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "4bbb0099cc26bfafdc173eb2fe3c7c80"
server_key_iv: "453ed0a15eaef767026604c376aba281"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.271247 26217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:55.274462 26217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:55.274670 26217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:55.274762 26217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:55.274909 26217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:55.275135 26217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:55.275164 26217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.275183 26217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:55.275197 26217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.280947 26217 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:35033
I20260925 14:53:55.281030 26344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:35033 every 8 connection(s)
I20260925 14:53:55.281338 26217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:55.284317 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.282087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:60313 (local address 127.20.199.62:34111)
0925 14:53:55.282149 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.282151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.282418 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:53:55.282850 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.282851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.282856 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.282908 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.284047 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.284049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.284049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.284053 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.284068 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.284071 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.284130 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.284226 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.284245 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:53:55.284528 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.282079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34111 (local address 127.20.199.3:60313)
0925 14:53:55.282283 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.282337 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.282770 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.282959 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.282962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.282970 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.283897 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.283897 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.283924 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.283951 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.284132 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.284134 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.284136 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.284229 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.284298 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260925 14:53:55.285071 26345 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34111
I20260925 14:53:55.285171 26345 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:55.285318 26345 heartbeater.cc:511] Master 127.20.199.62:34111 requested a full tablet report, sending...
I20260925 14:53:55.285712 25895 ts_manager.cc:194] Registered new tserver with Master: 09aa9b2879144383ab9eb07efd1322c6 (127.20.199.3:35033)
I20260925 14:53:55.291356 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26217
I20260925 14:53:55.291465 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:55.291651 21276 external_mini_cluster.cc:1428] Setting key 61912ab3e60c9585f63d1498d41656aa
I20260925 14:53:55.292533 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:55.294936 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.293856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.20.199.62:34111)
0925 14:53:55.293931 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.293933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.294026 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:53:55.294061 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.294062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.294066 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.294110 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.294721 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.294721 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.294722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.294725 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.294739 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.294742 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.294783 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.294819 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.294830 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:53:55.296839 25895 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:55.298494 25895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:55.302661 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.300550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:35033 (local address 127.0.0.1:44372)
0925 14:53:55.301430 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.301439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.301482 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.301908 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.301909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.301911 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.302272 (+ 361us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.302273 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.302279 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.302290 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.302510 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.302511 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.302512 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.302570 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.302583 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":102,"threads_started":1}
I20260925 14:53:55.302683 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.300722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.20.199.1:32919)
0925 14:53:55.300903 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.300908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.301180 (+ 272us) server_negotiation.cc:407] Connection header received
0925 14:53:55.301251 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.301255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.301267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.301349 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.302164 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.302172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.302173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.302178 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.302216 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.302237 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.302314 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.302585 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.302600 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260925 14:53:55.302665 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.300545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:32919 (local address 127.0.0.1:59498)
0925 14:53:55.301081 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.301111 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.301187 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.301369 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.301376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.301383 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.302042 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.302042 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.302059 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.302080 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.302491 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.302498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.302501 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.302574 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.302589 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":65,"threads_started":1}
I20260925 14:53:55.302906 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.300545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:33869 (local address 127.0.0.1:42260)
0925 14:53:55.301282 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.301291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.301334 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.302178 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.302179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.302182 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.302500 (+ 318us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.302501 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.302507 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.302515 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.302785 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.302786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.302786 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.302835 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.302848 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260925 14:53:55.303054 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.300780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.20.199.3:35033)
0925 14:53:55.300931 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.300936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.301491 (+ 555us) server_negotiation.cc:407] Connection header received
0925 14:53:55.301525 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.301528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.301539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.301616 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.302370 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.302377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.302377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.302382 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.302416 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.302441 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.302508 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.302940 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.302957 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260925 14:53:55.303352 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.300929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.20.199.2:33869)
0925 14:53:55.301081 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.301084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.301621 (+ 537us) server_negotiation.cc:407] Connection header received
0925 14:53:55.301649 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.301651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.301660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.301728 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.302600 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.302606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.302606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.302611 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.302643 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.302665 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.303260 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.303279 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.303292 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260925 14:53:55.303668 26017 tablet_service.cc:1467] Processing CreateTablet for tablet 27b7acc57d5340d793b0340195457ae0 (DEFAULT_TABLE table=test-table [id=b2f8dc1604f94da684ab1195220b56fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:55.304044 26017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b7acc57d5340d793b0340195457ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:55.303952 26148 tablet_service.cc:1467] Processing CreateTablet for tablet 27b7acc57d5340d793b0340195457ae0 (DEFAULT_TABLE table=test-table [id=b2f8dc1604f94da684ab1195220b56fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:55.304270 26148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b7acc57d5340d793b0340195457ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:55.306033 26279 tablet_service.cc:1467] Processing CreateTablet for tablet 27b7acc57d5340d793b0340195457ae0 (DEFAULT_TABLE table=test-table [id=b2f8dc1604f94da684ab1195220b56fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:55.306336 26279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b7acc57d5340d793b0340195457ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:55.306794 26359 tablet_bootstrap.cc:492] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: Bootstrap starting.
I20260925 14:53:55.307112 26360 tablet_bootstrap.cc:492] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: Bootstrap starting.
I20260925 14:53:55.307596 26359 tablet_bootstrap.cc:654] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:55.307756 26360 tablet_bootstrap.cc:654] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:55.308003 26359 log.cc:824] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:55.308055 26360 log.cc:824] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:55.308723 26359 tablet_bootstrap.cc:492] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: No bootstrap required, opened a new log
I20260925 14:53:55.308804 26360 tablet_bootstrap.cc:492] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: No bootstrap required, opened a new log
I20260925 14:53:55.308805 26359 ts_tablet_manager.cc:1397] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:55.308871 26360 ts_tablet_manager.cc:1397] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:55.309211 26363 tablet_bootstrap.cc:492] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Bootstrap starting.
I20260925 14:53:55.309906 26363 tablet_bootstrap.cc:654] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:55.310261 26363 log.cc:824] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:55.311180 26363 tablet_bootstrap.cc:492] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: No bootstrap required, opened a new log
I20260925 14:53:55.311123 26360 raft_consensus.cc:348] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.311236 26360 raft_consensus.cc:374] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:55.311264 26360 raft_consensus.cc:729] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badf65d103d14146a34fd1e9f6afde7b, State: Initialized, Role: FOLLOWER
I20260925 14:53:55.311391 26363 ts_tablet_manager.cc:1397] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:55.311383 26360 consensus_queue.cc:260] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.311509 26359 raft_consensus.cc:348] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.311632 26360 ts_tablet_manager.cc:1428] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:55.311642 26214 heartbeater.cc:503] Master 127.20.199.62:34111 was elected leader, sending a full tablet report...
I20260925 14:53:55.311672 26359 raft_consensus.cc:374] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:55.311718 26359 raft_consensus.cc:729] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92818dabf43c436d89bb78379b772dc7, State: Initialized, Role: FOLLOWER
I20260925 14:53:55.311863 26359 consensus_queue.cc:260] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.312155 26083 heartbeater.cc:503] Master 127.20.199.62:34111 was elected leader, sending a full tablet report...
I20260925 14:53:55.312145 26359 ts_tablet_manager.cc:1428] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:53:55.314209 26363 raft_consensus.cc:348] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.314339 26363 raft_consensus.cc:374] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:55.314360 26363 raft_consensus.cc:729] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09aa9b2879144383ab9eb07efd1322c6, State: Initialized, Role: FOLLOWER
I20260925 14:53:55.314477 26363 consensus_queue.cc:260] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.314733 26363 ts_tablet_manager.cc:1428] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:55.314779 26345 heartbeater.cc:503] Master 127.20.199.62:34111 was elected leader, sending a full tablet report...
W20260925 14:53:55.417443 26215 tablet.cc:2369] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:55.468577 26366 raft_consensus.cc:482] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:55.468708 26366 raft_consensus.cc:504] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.469055 26366 leader_election.cc:288] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers badf65d103d14146a34fd1e9f6afde7b (127.20.199.2:33869), 09aa9b2879144383ab9eb07efd1322c6 (127.20.199.3:35033)
I20260925 14:53:55.470471 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.469272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:33869 (local address 127.20.199.1:34945)
0925 14:53:55.469386 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.469414 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.469459 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.469615 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.469616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.469619 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.470073 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.470073 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.470080 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.470091 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.470313 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.470315 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.470316 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.470381 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.470398 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 14:53:55.470471 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.469355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:34945 (local address 127.20.199.2:33869)
0925 14:53:55.469432 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.469434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.469452 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:53:55.469543 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.469544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.469549 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.469611 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.470189 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.470191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.470191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.470197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.470218 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.470224 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.470316 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.470386 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.470401 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:53:55.470604 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.469429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:45337 (local address 127.20.199.3:35033)
0925 14:53:55.469470 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.469472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.469621 (+ 149us) server_negotiation.cc:407] Connection header received
0925 14:53:55.469706 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.469707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.469713 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.469795 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.470378 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.470380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.470380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.470385 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.470400 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.470408 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.470470 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.470526 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.470541 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:55.470618 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.469322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:35033 (local address 127.20.199.1:45337)
0925 14:53:55.469528 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.469537 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.469630 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.469815 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.469816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.469818 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:55.470250 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:55.470250 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:55.470257 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:55.470269 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:55.470468 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:55.470469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:55.470469 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 14:53:55.470526 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:53:55.470541 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":131,"threads_started":1}
I20260925 14:53:55.470983 26168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7acc57d5340d793b0340195457ae0" candidate_uuid: "92818dabf43c436d89bb78379b772dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badf65d103d14146a34fd1e9f6afde7b" is_pre_election: true
I20260925 14:53:55.471009 26299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7acc57d5340d793b0340195457ae0" candidate_uuid: "92818dabf43c436d89bb78379b772dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09aa9b2879144383ab9eb07efd1322c6" is_pre_election: true
I20260925 14:53:55.471136 26168 raft_consensus.cc:2454] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92818dabf43c436d89bb78379b772dc7 in term 0.
I20260925 14:53:55.471164 26299 raft_consensus.cc:2454] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92818dabf43c436d89bb78379b772dc7 in term 0.
I20260925 14:53:55.471442 25973 leader_election.cc:302] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09aa9b2879144383ab9eb07efd1322c6, 92818dabf43c436d89bb78379b772dc7; no voters:
I20260925 14:53:55.471637 26366 raft_consensus.cc:2781] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:55.471688 26366 raft_consensus.cc:482] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:55.471705 26366 raft_consensus.cc:3037] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:55.472457 26366 raft_consensus.cc:504] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.472622 26366 leader_election.cc:288] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [CANDIDATE]: Term 1 election: Requested vote from peers badf65d103d14146a34fd1e9f6afde7b (127.20.199.2:33869), 09aa9b2879144383ab9eb07efd1322c6 (127.20.199.3:35033)
I20260925 14:53:55.472761 26168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7acc57d5340d793b0340195457ae0" candidate_uuid: "92818dabf43c436d89bb78379b772dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badf65d103d14146a34fd1e9f6afde7b"
I20260925 14:53:55.472822 26299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b7acc57d5340d793b0340195457ae0" candidate_uuid: "92818dabf43c436d89bb78379b772dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09aa9b2879144383ab9eb07efd1322c6"
I20260925 14:53:55.472854 26168 raft_consensus.cc:3037] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:55.472900 26299 raft_consensus.cc:3037] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:55.473656 26168 raft_consensus.cc:2454] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92818dabf43c436d89bb78379b772dc7 in term 1.
I20260925 14:53:55.473659 26299 raft_consensus.cc:2454] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92818dabf43c436d89bb78379b772dc7 in term 1.
I20260925 14:53:55.473847 25973 leader_election.cc:302] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09aa9b2879144383ab9eb07efd1322c6, 92818dabf43c436d89bb78379b772dc7; no voters:
I20260925 14:53:55.473948 26366 raft_consensus.cc:2781] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:55.474128 26366 raft_consensus.cc:686] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [term 1 LEADER]: Becoming Leader. State: Replica: 92818dabf43c436d89bb78379b772dc7, State: Running, Role: LEADER
I20260925 14:53:55.474236 26366 consensus_queue.cc:237] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } }
I20260925 14:53:55.475131 25894 catalog_manager.cc:5495] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92818dabf43c436d89bb78379b772dc7 (127.20.199.1). New cstate: current_term: 1 leader_uuid: "92818dabf43c436d89bb78379b772dc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92818dabf43c436d89bb78379b772dc7" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:55.487106 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.486235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.20.199.1:32919)
0925 14:53:55.486285 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.486287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.486296 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:55.486360 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.486360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.486371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.486418 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.486898 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.486899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.486900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.486903 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.486917 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.486920 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.486965 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.487004 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.487014 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:53:55.488868 26168 raft_consensus.cc:1261] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 1 FOLLOWER]: Refusing update from remote peer 92818dabf43c436d89bb78379b772dc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:55.488868 26299 raft_consensus.cc:1261] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 1 FOLLOWER]: Refusing update from remote peer 92818dabf43c436d89bb78379b772dc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:55.489257 26369 consensus_queue.cc:1035] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09aa9b2879144383ab9eb07efd1322c6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:55.489428 26366 consensus_queue.cc:1035] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "badf65d103d14146a34fd1e9f6afde7b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:55.491724 26372 mvcc.cc:205] Tried to move back new op lower bound from 7333265553359474688 to 7333265553303212032. Current Snapshot: MvccSnapshot[applied={T|T < 7333265553359474688}]
I20260925 14:53:55.492187 26373 mvcc.cc:205] Tried to move back new op lower bound from 7333265553359474688 to 7333265553303212032. Current Snapshot: MvccSnapshot[applied={T|T < 7333265553359474688}]
I20260925 14:53:55.494606 25895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:55.494704 25895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:55.495381 25895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c83003e4486944e0bc643f82a5066519: Sending DeleteTablet for 3 replicas of tablet 27b7acc57d5340d793b0340195457ae0
I20260925 14:53:55.495718 26148 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b7acc57d5340d793b0340195457ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:55 UTC) from {username='slave'} at 127.0.0.1:42260
I20260925 14:53:55.495752 26279 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b7acc57d5340d793b0340195457ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:55 UTC) from {username='slave'} at 127.0.0.1:44372
I20260925 14:53:55.495986 26017 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b7acc57d5340d793b0340195457ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:55 UTC) from {username='slave'} at 127.0.0.1:59498
I20260925 14:53:55.496135 26381 tablet_replica.cc:330] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: stopping tablet replica
I20260925 14:53:55.496171 26383 tablet_replica.cc:330] T 27b7acc57d5340d793b0340195457ae0 P 92818dabf43c436d89bb78379b772dc7: stopping tablet replica
I20260925 14:53:55.496232 26381 raft_consensus.cc:2229] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:55.496287 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25955
W20260925 14:53:55.500929 25880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:32919 (error 108)
I20260925 14:53:55.496337 26381 raft_consensus.cc:2258] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:55.501349 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26086
I20260925 14:53:55.501375 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.501246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:32919 (local address 127.0.0.1:59510)
0925 14:53:55.501299 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.501342 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:53:55.501472 26382 tablet_replica.cc:330] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b: stopping tablet replica
I20260925 14:53:55.501570 26382 raft_consensus.cc:2229] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:55.501665 26382 raft_consensus.cc:2258] T 27b7acc57d5340d793b0340195457ae0 P badf65d103d14146a34fd1e9f6afde7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:55.502540 26381 ts_tablet_manager.cc:1905] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:55.504420 26381 ts_tablet_manager.cc:1918] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:55.504475 26381 log.cc:1196] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/27b7acc57d5340d793b0340195457ae0
I20260925 14:53:55.504660 26381 ts_tablet_manager.cc:1939] T 27b7acc57d5340d793b0340195457ae0 P 09aa9b2879144383ab9eb07efd1322c6: Deleting consensus metadata
I20260925 14:53:55.505012 25882 catalog_manager.cc:4841] TS 09aa9b2879144383ab9eb07efd1322c6 (127.20.199.3:35033): tablet 27b7acc57d5340d793b0340195457ae0 (table test-table [id=b2f8dc1604f94da684ab1195220b56fe]) successfully deleted
W20260925 14:53:55.505616 25880 catalog_manager.cc:4584] TS 92818dabf43c436d89bb78379b772dc7 (127.20.199.1:32919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27b7acc57d5340d793b0340195457ae0: Network error: Client connection negotiation failed: client connection to 127.20.199.1:32919: connect: Connection refused (error 111)
I20260925 14:53:55.506062 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26217
I20260925 14:53:55.506158 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.506015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:33869 (local address 127.0.0.1:42276)
0925 14:53:55.506076 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.506122 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.2:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:53:55.510690 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 25864
2026-09-25T14:53:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (752 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:53:55.529865 21276 mini_oidc.cc:152] Starting JWKS server
I20260925 14:53:55.530895 21276 webserver.cc:460] Webserver started at https://127.0.0.1:41443/ using document root <none> and password file <none>
I20260925 14:53:55.531010 21276 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:53:55.531181 21276 webserver.cc:460] Webserver started at http://127.0.0.1:42839/ using document root <none> and password file <none>
2026-09-25T14:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:55Z Disabled control of system clock
I20260925 14:53:55.538776 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:44447
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:44447
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/testchainca.cert with env {}
W20260925 14:53:55.616940 26390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:55.617136 26390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:55.617157 26390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:55.618266 26390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:55.618301 26390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:55.618314 26390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:55.618326 26390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:55.618338 26390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:53:55.618350 26390 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:53:55.618361 26390 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41443/jwks.json
I20260925 14:53:55.619836 26390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:44447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:55.620018 26390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:55.620174 26390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:55.624925 26395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.624950 26390 server_base.cc:1034] running on GCE node
W20260925 14:53:55.625056 26396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.625123 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.625464 26390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:55.625743 26390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:55.626915 26390 hybrid_clock.cc:648] HybridClock initialized: now 1790348035626891 us; error 44 us; skew 500 ppm
I20260925 14:53:55.628387 26390 webserver.cc:460] Webserver started at http://127.20.199.62:41377/ using document root <none> and password file <none>
I20260925 14:53:55.628667 26390 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:55.628736 26390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:55.628890 26390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:55.630043 26390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "85873360ec3d4db28672b8437c194a15"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "7e88fa8477a8e2de28a4ccff9c327e55"
server_key_iv: "70244148575d0def2a9c7b62a2563d5c"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.630453 26390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "85873360ec3d4db28672b8437c194a15"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "7e88fa8477a8e2de28a4ccff9c327e55"
server_key_iv: "70244148575d0def2a9c7b62a2563d5c"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.631896 26390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:55.632712 26404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.633013 26390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:55.633116 26390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "85873360ec3d4db28672b8437c194a15"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "7e88fa8477a8e2de28a4ccff9c327e55"
server_key_iv: "70244148575d0def2a9c7b62a2563d5c"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.633190 26390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:55.655434 26390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:55.655773 26390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:55.655929 26390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:55.659860 26390 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:44447
I20260925 14:53:55.659905 26456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:44447 every 8 connection(s)
I20260925 14:53:55.660259 26390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:55.661218 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:55.662853 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26390
I20260925 14:53:55.662972 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:55.663199 21276 external_mini_cluster.cc:1428] Setting key 54a2d0ae5d82c8f4028ee6d5b618547f
I20260925 14:53:55.664086 26457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15: Bootstrap starting.
I20260925 14:53:55.664733 26457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:55.665087 26457 log.cc:824] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:55.665833 26457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15: No bootstrap required, opened a new log
I20260925 14:53:55.667618 26457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85873360ec3d4db28672b8437c194a15" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44447 } }
I20260925 14:53:55.667757 26457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:55.667804 26457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85873360ec3d4db28672b8437c194a15, State: Initialized, Role: FOLLOWER
I20260925 14:53:55.667948 26457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85873360ec3d4db28672b8437c194a15" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44447 } }
I20260925 14:53:55.668044 26457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:55.668099 26457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:55.668165 26457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:55.668771 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.663812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.20.199.62:44447)
0925 14:53:55.664010 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.664018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.664043 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:53:55.664492 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.664500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.664561 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.664685 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.664955 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.665935 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.666706 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.667118 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.667265 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:55.668018 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.668022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.668022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.668239 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.668285 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.668300 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.668378 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.668452 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.668533 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260925 14:53:55.668902 26457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85873360ec3d4db28672b8437c194a15" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44447 } }
I20260925 14:53:55.669009 26457 leader_election.cc:302] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85873360ec3d4db28672b8437c194a15; no voters:
I20260925 14:53:55.669203 26457 leader_election.cc:288] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:55.669266 26462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:55.669430 26462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [term 1 LEADER]: Becoming Leader. State: Replica: 85873360ec3d4db28672b8437c194a15, State: Running, Role: LEADER
I20260925 14:53:55.669489 26457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:55.669525 26462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85873360ec3d4db28672b8437c194a15" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44447 } }
I20260925 14:53:55.669842 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:55.669886 26463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85873360ec3d4db28672b8437c194a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85873360ec3d4db28672b8437c194a15" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44447 } } }
I20260925 14:53:55.669970 26463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:55.670189 26469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:55.670689 26469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:55.671154 26464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85873360ec3d4db28672b8437c194a15. Latest consensus state: current_term: 1 leader_uuid: "85873360ec3d4db28672b8437c194a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85873360ec3d4db28672b8437c194a15" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44447 } } }
I20260925 14:53:55.671253 26464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:55.672569 26469 catalog_manager.cc:1347] Generated new cluster ID: ee8e2fb26d03418c97f5858f20ce773e
I20260925 14:53:55.672636 26469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:55.691882 26469 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:55.692410 26469 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:55.696244 26469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85873360ec3d4db28672b8437c194a15: Generated new TSK 0
I20260925 14:53:55.696426 26469 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:53:55.730854 26487 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:53:55.731024 26487 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:53:55.731242 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.708304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.20.199.62:44447)
0925 14:53:55.708373 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.708375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.708543 (+ 168us) server_negotiation.cc:407] Connection header received
0925 14:53:55.708668 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.708669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.708680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.708745 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:53:55.708923 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.709490 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.710683 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.710892 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.710950 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:55.711171 (+ 221us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:53:55.730859 (+ 19688us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:53:55.731173 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 14:53:55.731328 26460 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 14:53:55.731467 26486 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:55.708236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:44447 (local address 127.0.0.1:55262)
0925 14:53:55.708459 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.708472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.708555 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.708771 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.708772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.708774 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:53:55.708848 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:55.708850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.709607 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.709608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:55.710569 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:55.710572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.710974 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.710975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:55.711042 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:55.711064 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:53:55.731003 (+ 19939us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:53:55.731117 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.199.62:44447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.199.62:44447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 14:53:55.731907 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26390
2026-09-25T14:53:55Z chronyd exiting
I20260925 14:53:55.768075 21276 test_util.cc:175] -----------------------------------------------
I20260925 14:53:55.768160 21276 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348014035135-21276-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:53:55.768940 21276 mini_oidc.cc:152] Starting JWKS server
I20260925 14:53:55.769692 21276 webserver.cc:460] Webserver started at https://127.0.0.1:44589/ using document root <none> and password file <none>
I20260925 14:53:55.769733 21276 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:53:55.769930 21276 webserver.cc:460] Webserver started at http://127.0.0.1:35117/ using document root <none> and password file <none>
2026-09-25T14:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:55Z Disabled control of system clock
I20260925 14:53:55.777505 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34021
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34021
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44589/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/testchainca.cert with env {}
W20260925 14:53:55.860188 26493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:55.860428 26493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:55.860464 26493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:55.861644 26493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:55.861711 26493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:55.861730 26493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:55.861750 26493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:55.861786 26493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:53:55.861814 26493 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:53:55.861840 26493 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44589/jwks.json
I20260925 14:53:55.863346 26493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44589/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:55.863618 26493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:55.863840 26493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:55.868707 26493 server_base.cc:1034] running on GCE node
W20260925 14:53:55.868687 26501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.868654 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:55.868676 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:55.869158 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:55.869518 26493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:55.870688 26493 hybrid_clock.cc:648] HybridClock initialized: now 1790348035870676 us; error 38 us; skew 500 ppm
I20260925 14:53:55.871966 26493 webserver.cc:460] Webserver started at http://127.20.199.62:42533/ using document root <none> and password file <none>
I20260925 14:53:55.872218 26493 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:55.872293 26493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:55.872442 26493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:55.873611 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "19821658a0374abdbcd87fd0f86a7cff"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "ce2bcd319118bef29e459a98d45f0407"
server_key_iv: "8acf7ae24f569f10df27cee5beef0921"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.873973 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "19821658a0374abdbcd87fd0f86a7cff"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "ce2bcd319118bef29e459a98d45f0407"
server_key_iv: "8acf7ae24f569f10df27cee5beef0921"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.875432 26493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:55.876163 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:55.876340 26493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:55.876441 26493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "19821658a0374abdbcd87fd0f86a7cff"
format_stamp: "Formatted at 2026-09-25 14:53:55 on dist-test-slave-8qt6"
server_key: "ce2bcd319118bef29e459a98d45f0407"
server_key_iv: "8acf7ae24f569f10df27cee5beef0921"
server_key_version: "encryptionkey@0"
I20260925 14:53:55.876510 26493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:55.922555 26493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:55.922924 26493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:55.923081 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:55.927048 26493 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34021
I20260925 14:53:55.927085 26559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34021 every 8 connection(s)
I20260925 14:53:55.927450 26493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:55.928092 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:55.930749 26560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff: Bootstrap starting.
I20260925 14:53:55.931303 26560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:55.931526 26560 log.cc:824] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:55.931984 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26493
I20260925 14:53:55.932090 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:55.932267 21276 external_mini_cluster.cc:1428] Setting key e401e71bbb3294d8b46fb0b2fe752e2d
I20260925 14:53:55.932260 26560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff: No bootstrap required, opened a new log
I20260925 14:53:55.935359 26560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19821658a0374abdbcd87fd0f86a7cff" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34021 } }
I20260925 14:53:55.935527 26560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:55.935577 26560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19821658a0374abdbcd87fd0f86a7cff, State: Initialized, Role: FOLLOWER
I20260925 14:53:55.935708 26560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19821658a0374abdbcd87fd0f86a7cff" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34021 } }
I20260925 14:53:55.935810 26560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:55.935853 26560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:55.935910 26560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:55.936508 26560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19821658a0374abdbcd87fd0f86a7cff" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34021 } }
I20260925 14:53:55.936635 26560 leader_election.cc:302] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19821658a0374abdbcd87fd0f86a7cff; no voters:
I20260925 14:53:55.936815 26560 leader_election.cc:288] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:55.937021 26566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:55.937134 26560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:55.937160 26566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [term 1 LEADER]: Becoming Leader. State: Replica: 19821658a0374abdbcd87fd0f86a7cff, State: Running, Role: LEADER
I20260925 14:53:55.937283 26566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19821658a0374abdbcd87fd0f86a7cff" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34021 } }
I20260925 14:53:55.937647 26566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19821658a0374abdbcd87fd0f86a7cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19821658a0374abdbcd87fd0f86a7cff" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34021 } } }
I20260925 14:53:55.937724 26566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:55.937866 26566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19821658a0374abdbcd87fd0f86a7cff. Latest consensus state: current_term: 1 leader_uuid: "19821658a0374abdbcd87fd0f86a7cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19821658a0374abdbcd87fd0f86a7cff" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34021 } } }
I20260925 14:53:55.937934 26566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:55.938436 26571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:55.938957 26571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:55.941361 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.932812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.20.199.62:34021)
0925 14:53:55.933129 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.933142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.933186 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:53:55.933925 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.933935 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.934023 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.934181 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:55.934495 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.936663 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.938465 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.939115 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.939226 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:55.940617 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:55.940621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:55.940621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:55.940821 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:55.940855 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:55.940866 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:55.940923 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:55.941085 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:53:55.941178 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260925 14:53:55.941632 26571 catalog_manager.cc:1347] Generated new cluster ID: 5f281657d066418f8de6a80b5b6d5354
I20260925 14:53:55.941692 26571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:55.942551 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:55.952795 26571 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:55.953444 26571 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:55.961083 26571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19821658a0374abdbcd87fd0f86a7cff: Generated new TSK 0
I20260925 14:53:55.961258 26571 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:53:55.988175 26590 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:53:55.988282 26590 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:53:55.988565 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:55.974116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.20.199.62:34021)
0925 14:53:55.974163 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:55.974165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:55.974243 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:53:55.974331 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:55.974332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:55.974345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:55.974427 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:53:55.974553 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.975364 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.976102 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.976404 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.976473 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:55.976603 (+ 130us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:53:55.988336 (+ 11733us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:53:55.988491 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:53:55.988628 26564 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 14:53:55.988631 26589 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:55.974043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34021 (local address 127.0.0.1:56588)
0925 14:53:55.974185 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:55.974195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:55.974257 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:55.974431 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:55.974432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:55.974433 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:53:55.974482 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:55.974484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.975441 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.975442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:55.976024 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:55.976026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:55.976480 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:55.976481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:55.976521 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:55.976525 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:53:55.988448 (+ 11923us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:53:55.988545 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.199.62:34021: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 14:53:55.988994 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26493
2026-09-25T14:53:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (257 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:53:56.026479 21276 mini_oidc.cc:152] Starting JWKS server
I20260925 14:53:56.027160 21276 webserver.cc:460] Webserver started at https://127.0.0.1:37169/ using document root <none> and password file <none>
I20260925 14:53:56.027206 21276 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:53:56.027340 21276 webserver.cc:460] Webserver started at http://127.0.0.1:36617/ using document root <none> and password file <none>
2026-09-25T14:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:56Z Disabled control of system clock
I20260925 14:53:56.034710 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46431
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:46431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37169/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:53:56.114200 26596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:56.114441 26596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:56.114473 26596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:56.115610 26596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:56.115671 26596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:56.115693 26596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:56.115723 26596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:56.115749 26596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:53:56.115777 26596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:53:56.115800 26596 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37169/jwks.json
I20260925 14:53:56.117126 26596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:46431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37169/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:56.117442 26596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:56.117715 26596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:56.122324 26601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:56.122315 26604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:56.122381 26602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:56.122593 26596 server_base.cc:1034] running on GCE node
I20260925 14:53:56.122843 26596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:56.123106 26596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:56.124265 26596 hybrid_clock.cc:648] HybridClock initialized: now 1790348036124256 us; error 41 us; skew 500 ppm
I20260925 14:53:56.125339 26596 webserver.cc:460] Webserver started at http://127.20.199.62:44611/ using document root <none> and password file <none>
I20260925 14:53:56.125576 26596 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:56.125640 26596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:56.125794 26596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:56.126847 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "14a05db57de24d0f94af8e7cbbafb098"
format_stamp: "Formatted at 2026-09-25 14:53:56 on dist-test-slave-8qt6"
server_key: "d69993a59c7a98d47e593f3afa112dc4"
server_key_iv: "d16c2f1b1a7090e815adf6ab548f02c4"
server_key_version: "encryptionkey@0"
I20260925 14:53:56.127200 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "14a05db57de24d0f94af8e7cbbafb098"
format_stamp: "Formatted at 2026-09-25 14:53:56 on dist-test-slave-8qt6"
server_key: "d69993a59c7a98d47e593f3afa112dc4"
server_key_iv: "d16c2f1b1a7090e815adf6ab548f02c4"
server_key_version: "encryptionkey@0"
I20260925 14:53:56.128439 26596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:56.129078 26610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:56.129237 26596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:56.129331 26596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "14a05db57de24d0f94af8e7cbbafb098"
format_stamp: "Formatted at 2026-09-25 14:53:56 on dist-test-slave-8qt6"
server_key: "d69993a59c7a98d47e593f3afa112dc4"
server_key_iv: "d16c2f1b1a7090e815adf6ab548f02c4"
server_key_version: "encryptionkey@0"
I20260925 14:53:56.129424 26596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:56.143748 26596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:56.144038 26596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:56.144174 26596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:56.147943 26596 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:46431
I20260925 14:53:56.147974 26662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:46431 every 8 connection(s)
I20260925 14:53:56.148298 26596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:56.148653 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26596
I20260925 14:53:56.148756 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:56.148967 21276 external_mini_cluster.cc:1428] Setting key fcb3b98fb650b2fe54731510d03b07ee
I20260925 14:53:56.149049 26663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:56.152426 26663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098: Bootstrap starting.
I20260925 14:53:56.153019 26663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:56.153303 26663 log.cc:824] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:56.154083 26663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098: No bootstrap required, opened a new log
I20260925 14:53:56.155090 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:56.149766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.20.199.62:46431)
0925 14:53:56.149942 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:56.149951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:56.149982 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:53:56.150444 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:56.150453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:56.150531 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:56.150681 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:56.150858 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.152219 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.153010 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.153623 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.153711 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:56.154421 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:56.154426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:56.154426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:56.154622 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:56.154664 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:56.154677 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:56.154745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:56.154799 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:53:56.154876 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260925 14:53:56.156078 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:56.156605 26663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a05db57de24d0f94af8e7cbbafb098" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46431 } }
I20260925 14:53:56.156788 26663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:56.156837 26663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a05db57de24d0f94af8e7cbbafb098, State: Initialized, Role: FOLLOWER
I20260925 14:53:56.156967 26663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a05db57de24d0f94af8e7cbbafb098" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46431 } }
I20260925 14:53:56.157049 26663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:56.157081 26663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:56.157130 26663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:56.157735 26663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a05db57de24d0f94af8e7cbbafb098" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46431 } }
I20260925 14:53:56.157856 26663 leader_election.cc:302] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a05db57de24d0f94af8e7cbbafb098; no voters:
I20260925 14:53:56.158003 26663 leader_election.cc:288] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:56.158046 26668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:56.158200 26668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [term 1 LEADER]: Becoming Leader. State: Replica: 14a05db57de24d0f94af8e7cbbafb098, State: Running, Role: LEADER
I20260925 14:53:56.158244 26663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:56.158298 26668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a05db57de24d0f94af8e7cbbafb098" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46431 } }
I20260925 14:53:56.158660 26669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a05db57de24d0f94af8e7cbbafb098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a05db57de24d0f94af8e7cbbafb098" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46431 } } }
I20260925 14:53:56.158682 26670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a05db57de24d0f94af8e7cbbafb098. Latest consensus state: current_term: 1 leader_uuid: "14a05db57de24d0f94af8e7cbbafb098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a05db57de24d0f94af8e7cbbafb098" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46431 } } }
I20260925 14:53:56.158808 26669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:56.158828 26670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:56.159212 26678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:56.159749 26678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:56.161234 26678 catalog_manager.cc:1347] Generated new cluster ID: b1cebefba73840d3beb781c02d7a9c19
I20260925 14:53:56.161288 26678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:56.207180 26678 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:56.207914 26678 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:56.215368 26678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a05db57de24d0f94af8e7cbbafb098: Generated new TSK 0
I20260925 14:53:56.215574 26678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:56.226871 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:56.223230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.20.199.62:46431)
0925 14:53:56.223288 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:56.223290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:56.223396 (+ 106us) server_negotiation.cc:407] Connection header received
0925 14:53:56.223452 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:56.223453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:56.223466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:56.223540 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:53:56.223642 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.224412 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.225167 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.225516 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.225587 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:56.225726 (+ 139us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:53:56.226659 (+ 933us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:53:56.226800 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58330: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 14:53:56.226933 26666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58330: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 14:53:56.228819 26692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:56.223153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46431 (local address 127.0.0.1:58330)
0925 14:53:56.223336 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:56.223344 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:56.223395 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:56.223554 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:56.223554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:56.223556 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:53:56.223603 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:56.223605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.224500 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.224500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:56.225091 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:56.225093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.225590 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.225591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:56.225631 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:56.225635 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:53:56.226748 (+ 1113us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:53:56.228724 (+ 1976us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.199.62:46431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":1786,"thread_start_us":108,"threads_started":1}
I20260925 14:53:56.229131 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26596
E20260925 14:53:56.238159 26693 webserver.cc:533] Webserver: sslize failed:
E20260925 14:53:56.238225 26693 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T14:53:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:56Z Disabled control of system clock
I20260925 14:53:56.274040 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:33883
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:33883
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/testchainca.cert with env {}
W20260925 14:53:56.351213 26697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:56.351446 26697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:56.351482 26697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:56.352612 26697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:56.352675 26697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:56.352696 26697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:56.352725 26697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:56.352754 26697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:53:56.352785 26697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:53:56.352829 26697 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:53:56.354293 26697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:33883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:56.354549 26697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:56.354755 26697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:56.359656 26702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:56.359694 26697 server_base.cc:1034] running on GCE node
W20260925 14:53:56.359874 26705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:56.359654 26703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:56.360199 26697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:56.360538 26697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:56.361685 26697 hybrid_clock.cc:648] HybridClock initialized: now 1790348036361681 us; error 46 us; skew 500 ppm
I20260925 14:53:56.362941 26697 webserver.cc:460] Webserver started at http://127.20.199.62:46637/ using document root <none> and password file <none>
I20260925 14:53:56.363142 26697 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:56.363183 26697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:56.363267 26697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:56.364205 26697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "8843a0a1c0ca481995bec88f9a5e1932"
format_stamp: "Formatted at 2026-09-25 14:53:56 on dist-test-slave-8qt6"
server_key: "358163e1fd970f1c3093e6d0d7361610"
server_key_iv: "761aa144ecbf74cc0e9a742a1c03fd2d"
server_key_version: "encryptionkey@0"
I20260925 14:53:56.364490 26697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "8843a0a1c0ca481995bec88f9a5e1932"
format_stamp: "Formatted at 2026-09-25 14:53:56 on dist-test-slave-8qt6"
server_key: "358163e1fd970f1c3093e6d0d7361610"
server_key_iv: "761aa144ecbf74cc0e9a742a1c03fd2d"
server_key_version: "encryptionkey@0"
I20260925 14:53:56.365797 26697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:56.366438 26711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:56.366600 26697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:56.366695 26697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "8843a0a1c0ca481995bec88f9a5e1932"
format_stamp: "Formatted at 2026-09-25 14:53:56 on dist-test-slave-8qt6"
server_key: "358163e1fd970f1c3093e6d0d7361610"
server_key_iv: "761aa144ecbf74cc0e9a742a1c03fd2d"
server_key_version: "encryptionkey@0"
I20260925 14:53:56.366772 26697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:56.384212 26697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:56.384531 26697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:56.384680 26697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:56.388523 26697 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:33883
I20260925 14:53:56.388571 26763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:33883 every 8 connection(s)
I20260925 14:53:56.388918 26697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:56.389842 26764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:56.391988 26764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932: Bootstrap starting.
I20260925 14:53:56.392544 26764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:56.392829 26764 log.cc:824] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:56.393471 26764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932: No bootstrap required, opened a new log
I20260925 14:53:56.394994 26764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8843a0a1c0ca481995bec88f9a5e1932" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33883 } }
I20260925 14:53:56.395123 26764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:56.395179 26764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8843a0a1c0ca481995bec88f9a5e1932, State: Initialized, Role: FOLLOWER
I20260925 14:53:56.395339 26764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8843a0a1c0ca481995bec88f9a5e1932" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33883 } }
I20260925 14:53:56.395399 26764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:56.395434 26764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:56.395481 26764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:56.395984 26764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8843a0a1c0ca481995bec88f9a5e1932" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33883 } }
I20260925 14:53:56.396093 26764 leader_election.cc:302] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8843a0a1c0ca481995bec88f9a5e1932; no voters:
I20260925 14:53:56.396260 26764 leader_election.cc:288] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:56.396315 26767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:56.396461 26767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [term 1 LEADER]: Becoming Leader. State: Replica: 8843a0a1c0ca481995bec88f9a5e1932, State: Running, Role: LEADER
I20260925 14:53:56.396499 26764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:56.396548 26767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8843a0a1c0ca481995bec88f9a5e1932" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33883 } }
I20260925 14:53:56.396930 26769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8843a0a1c0ca481995bec88f9a5e1932. Latest consensus state: current_term: 1 leader_uuid: "8843a0a1c0ca481995bec88f9a5e1932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8843a0a1c0ca481995bec88f9a5e1932" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33883 } } }
I20260925 14:53:56.397017 26768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8843a0a1c0ca481995bec88f9a5e1932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8843a0a1c0ca481995bec88f9a5e1932" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33883 } } }
I20260925 14:53:56.397090 26768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:56.397013 26769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:56.397555 26775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:56.398010 26775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:56.398375 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26697
I20260925 14:53:56.398483 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:56.398672 21276 external_mini_cluster.cc:1428] Setting key 1fab49cbd7bd25361ab9ccfafd1c3c3a
I20260925 14:53:56.399554 26775 catalog_manager.cc:1347] Generated new cluster ID: b88202b25f3e4776ab143edca16b7de4
I20260925 14:53:56.399606 26775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:56.404508 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:56.399101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.20.199.62:33883)
0925 14:53:56.399294 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:56.399303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:56.399406 (+ 103us) server_negotiation.cc:407] Connection header received
0925 14:53:56.399565 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:56.399573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:56.399640 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:56.399784 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:56.399974 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.401363 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.402414 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.403036 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.403174 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:56.403811 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:56.403816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:56.403817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:56.404027 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:56.404067 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:56.404081 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:56.404151 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:56.404213 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:53:56.404301 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 14:53:56.405632 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:56.407922 26775 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:56.408406 26775 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:56.412554 26775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8843a0a1c0ca481995bec88f9a5e1932: Generated new TSK 0
I20260925 14:53:56.412711 26775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:56.850991 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:56.420047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.20.199.62:33883)
0925 14:53:56.420105 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:56.420108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:56.420228 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:53:56.420301 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:56.420302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:56.420312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:56.420369 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:53:56.420510 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.420954 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.422120 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.422300 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.422350 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:56.422521 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:53:56.850757 (+428236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:53:56.850932 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47522: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 14:53:56.851080 26787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47522: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 14:53:56.851164 26793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:56.419993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:33883 (local address 127.0.0.1:47522)
0925 14:53:56.420155 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:56.420167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:56.420233 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:56.420383 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:56.420384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:56.420386 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:53:56.420446 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:56.420448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.421037 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.421038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:56.422040 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:56.422043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:56.422372 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:56.422373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:56.422432 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:56.422438 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:53:56.850922 (+428484us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:53:56.851048 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.199.62:33883: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260925 14:53:56.851603 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26697
2026-09-25T14:53:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (602 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:57 dist-test-slave-8qt6 krb5kdc[26805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:57 dist-test-slave-8qt6 krb5kdc[26805](info): set up 2 sockets
Sep 25 14:53:57 dist-test-slave-8qt6 krb5kdc[26805](info): commencing operation
krb5kdc: starting...
W20260925 14:53:58.061527 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:53:58.179946 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:41949
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:58.268026 26818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:58.268333 26818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:58.268385 26818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:58.270323 26818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:58.270385 26818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:58.270411 26818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:58.270442 26818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:58.270468 26818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:58.272555 26818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:58.272825 26818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:58.273065 26818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:58.279034 26823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:58.279181 26818 server_base.cc:1034] running on GCE node
W20260925 14:53:58.279346 26826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:58.279035 26824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:58.279721 26818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:58.280041 26818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:58.281198 26818 hybrid_clock.cc:648] HybridClock initialized: now 1790348038281182 us; error 37 us; skew 500 ppm
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.284277 26818 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:53:58.284772 26818 webserver.cc:460] Webserver started at http://127.20.199.62:39451/ using document root <none> and password file <none>
I20260925 14:53:58.285029 26818 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:58.285130 26818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:58.285259 26818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:58.286769 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "370d43e07d734f479098ad4e80afe925"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "0cdf594d2f247c8bddadca1f7d4f318e"
server_key_iv: "153e057b062b844b296303a18853b033"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.287205 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "370d43e07d734f479098ad4e80afe925"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "0cdf594d2f247c8bddadca1f7d4f318e"
server_key_iv: "153e057b062b844b296303a18853b033"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.289078 26818 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:53:58.290894 26833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.291121 26818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:58.291222 26818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "370d43e07d734f479098ad4e80afe925"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "0cdf594d2f247c8bddadca1f7d4f318e"
server_key_iv: "153e057b062b844b296303a18853b033"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.291308 26818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:58.302977 26818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:58.305738 26818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:58.305852 26818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:58.309852 26818 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:41949
I20260925 14:53:58.309883 26885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:41949 every 8 connection(s)
I20260925 14:53:58.310222 26818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:58.311147 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:58.313390 26886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925: Bootstrap starting.
I20260925 14:53:58.314013 26886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:58.314298 26886 log.cc:824] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:58.314664 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26818
I20260925 14:53:58.314779 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:53:58.314970 21276 external_mini_cluster.cc:1428] Setting key 26f57367050e56a1f787e03557651ba4
I20260925 14:53:58.315037 26886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925: No bootstrap required, opened a new log
I20260925 14:53:58.317389 26886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370d43e07d734f479098ad4e80afe925" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41949 } }
I20260925 14:53:58.317605 26886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:58.317649 26886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370d43e07d734f479098ad4e80afe925, State: Initialized, Role: FOLLOWER
I20260925 14:53:58.317773 26886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370d43e07d734f479098ad4e80afe925" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41949 } }
I20260925 14:53:58.317859 26886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:58.317902 26886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:58.317950 26886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:58.318486 26886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370d43e07d734f479098ad4e80afe925" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41949 } }
I20260925 14:53:58.318600 26886 leader_election.cc:302] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 370d43e07d734f479098ad4e80afe925; no voters:
I20260925 14:53:58.318755 26886 leader_election.cc:288] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:58.318787 26891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:58.318907 26891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [term 1 LEADER]: Becoming Leader. State: Replica: 370d43e07d734f479098ad4e80afe925, State: Running, Role: LEADER
I20260925 14:53:58.318987 26891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370d43e07d734f479098ad4e80afe925" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41949 } }
I20260925 14:53:58.319020 26886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:58.319281 26891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370d43e07d734f479098ad4e80afe925. Latest consensus state: current_term: 1 leader_uuid: "370d43e07d734f479098ad4e80afe925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370d43e07d734f479098ad4e80afe925" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41949 } } }
I20260925 14:53:58.319360 26891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:58.319293 26893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "370d43e07d734f479098ad4e80afe925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370d43e07d734f479098ad4e80afe925" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41949 } } }
I20260925 14:53:58.319496 26893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925 [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:58.321043 26906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:58.321097 26906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:58.321446 26896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:58.321635 26896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:58.323176 26896 catalog_manager.cc:1347] Generated new cluster ID: 657a53f00e334846b90186cf95d0043a
I20260925 14:53:58.323225 26896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:58.326855 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.315471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.20.199.62:41949)
0925 14:53:58.315651 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.315658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.315683 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:53:58.316326 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.316333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.316382 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.316496 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.317299 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.318605 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.319638 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.320041 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.320166 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.323058 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.323063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.323063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.323075 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.325062 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.325476 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.325479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.325480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.325585 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.325781 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.325782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.325783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.326087 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.326249 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.326328 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.326469 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260925 14:53:58.349542 26896 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:58.350095 26896 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:58.355770 26896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 370d43e07d734f479098ad4e80afe925: Generated new TSK 0
I20260925 14:53:58.355962 26896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:53:58.385368 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:41949
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:58.465867 26912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:58.466102 26912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:58.466144 26912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:58.467312 26912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:58.467376 26912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:58.467418 26912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:53:58.468961 26912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:58.469244 26912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:58.469520 26912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:58.474459 26918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:58.474460 26920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:58.474598 26912 server_base.cc:1034] running on GCE node
W20260925 14:53:58.474692 26917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:58.474917 26912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:58.475108 26912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:58.476243 26912 hybrid_clock.cc:648] HybridClock initialized: now 1790348038476225 us; error 34 us; skew 500 ppm
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.478804 26912 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:53:58.479240 26912 webserver.cc:460] Webserver started at http://127.20.199.1:39035/ using document root <none> and password file <none>
I20260925 14:53:58.479506 26912 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:58.479559 26912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:58.479720 26912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:58.480742 26912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "9730dabd08484eae9a8de0cd35f27f45"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "b469f13424b70151c7221928079484c1"
server_key_iv: "f90ce7237f6c62a49dd30345088aff16"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.481086 26912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "9730dabd08484eae9a8de0cd35f27f45"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "b469f13424b70151c7221928079484c1"
server_key_iv: "f90ce7237f6c62a49dd30345088aff16"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.482434 26912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:58.483199 26927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.483426 26912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:58.483517 26912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "9730dabd08484eae9a8de0cd35f27f45"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "b469f13424b70151c7221928079484c1"
server_key_iv: "f90ce7237f6c62a49dd30345088aff16"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.483599 26912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:58.508883 26912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:58.511108 26912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:58.511288 26912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:58.511525 26912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:58.511825 26912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:58.511880 26912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.511920 26912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:58.511948 26912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.517555 26912 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:44163
I20260925 14:53:58.517602 27040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:44163 every 8 connection(s)
I20260925 14:53:58.517912 26912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:58.520289 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 26912
I20260925 14:53:58.520394 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:58.520571 21276 external_mini_cluster.cc:1428] Setting key 9e43db1e0e9d2b7bed0833022dbeaeeb
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:58.527870 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.518685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:60703 (local address 127.20.199.62:41949)
0925 14:53:58.518757 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.518759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.518846 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:53:58.519852 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.519853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.519865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.519946 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.520646 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.521352 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.522364 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.522586 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.522682 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.525071 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.525072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.525072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.525080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.526472 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.526953 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.526954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.526954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.526999 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.527273 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.527274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.527274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.527368 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.527414 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.527746 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.527807 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:53:58.528136 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.518568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:41949 (local address 127.20.199.1:60703)
0925 14:53:58.518723 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:58.518764 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:58.519746 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:58.520002 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:58.520006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:58.520295 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:58.520535 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:58.520540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.521470 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.521471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:58.522274 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:58.522276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.522677 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.522678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:58.522813 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.523311 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:58.523319 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:58.524978 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:58.526561 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:58.526567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:58.526569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:58.526880 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:58.527058 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:58.527059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:58.527059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:58.527205 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:58.527429 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:58.527435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:58.527612 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:53:58.527736 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:53:58.527903 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":96,"threads_started":1}
I20260925 14:53:58.528694 27041 heartbeater.cc:347] Connected to a master server at 127.20.199.62:41949
I20260925 14:53:58.528782 27041 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:58.528965 27041 heartbeater.cc:511] Master 127.20.199.62:41949 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
I20260925 14:53:58.529500 26850 ts_manager.cc:194] Registered new tserver with Master: 9730dabd08484eae9a8de0cd35f27f45 (127.20.199.1:44163)
I20260925 14:53:58.530094 26850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:60703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:53:58.547926 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:41949
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:58.627558 27046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:58.627810 27046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:58.627846 27046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:58.629032 27046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:58.629107 27046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:58.629153 27046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:53:58.630681 27046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:58.630977 27046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:58.631222 27046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:58.636273 27051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:58.636296 27052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:58.636322 27054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:58.636600 27046 server_base.cc:1034] running on GCE node
I20260925 14:53:58.636826 27046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:58.637086 27046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:58.638268 27046 hybrid_clock.cc:648] HybridClock initialized: now 1790348038638261 us; error 37 us; skew 500 ppm
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.640726 27046 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:53:58.641161 27046 webserver.cc:460] Webserver started at http://127.20.199.2:43493/ using document root <none> and password file <none>
I20260925 14:53:58.641439 27046 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:58.641515 27046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:58.641670 27046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:58.642781 27046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "155d8694981a484db1bc03968535de49"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "4fb36248e31ee46c9ccfd94e09338571"
server_key_iv: "1c3a2dba68bf389c77430abd1802fd7d"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.643146 27046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "155d8694981a484db1bc03968535de49"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "4fb36248e31ee46c9ccfd94e09338571"
server_key_iv: "1c3a2dba68bf389c77430abd1802fd7d"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.644498 27046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:58.645229 27061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.645442 27046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:58.645545 27046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "155d8694981a484db1bc03968535de49"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "4fb36248e31ee46c9ccfd94e09338571"
server_key_iv: "1c3a2dba68bf389c77430abd1802fd7d"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.645634 27046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:58.661684 27046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:58.663698 27046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:58.663856 27046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:58.664091 27046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:58.664402 27046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:58.664453 27046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.664494 27046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:58.664522 27046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.670449 27046 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:41977
I20260925 14:53:58.670565 27174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:41977 every 8 connection(s)
I20260925 14:53:58.670869 27046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:58.672569 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27046
I20260925 14:53:58.672658 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:58.672848 21276 external_mini_cluster.cc:1428] Setting key 65994862c934ce46b6e5f3642319af5b
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:53:58.681051 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.671636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:51893 (local address 127.20.199.62:41949)
0925 14:53:58.671679 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.671681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.671887 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:53:58.672970 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.672971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.672985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.673031 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.673780 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.674272 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.675686 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.675877 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.675934 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.678281 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.678282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.678282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.678290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.679663 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.680102 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.680103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.680103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.680154 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.680403 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.680404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.680404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.680518 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.680586 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.680876 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.680961 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:58.681275 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.671599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:41949 (local address 127.20.199.2:51893)
0925 14:53:58.671773 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:58.671808 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:58.672851 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:58.673099 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:58.673103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:58.673492 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:58.673682 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:58.673687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.674358 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.674359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:58.675593 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:58.675597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.675958 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.675959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:58.676101 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.676569 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:58.676579 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:58.678179 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:58.679755 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:58.679762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:58.679764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:58.679986 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:58.680222 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:58.680223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:58.680223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:58.680337 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:58.680596 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:58.680602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:58.680733 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 14:53:58.680830 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:53:58.680986 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260925 14:53:58.681747 27175 heartbeater.cc:347] Connected to a master server at 127.20.199.62:41949
I20260925 14:53:58.681845 27175 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:58.681988 27175 heartbeater.cc:511] Master 127.20.199.62:41949 requested a full tablet report, sending...
I20260925 14:53:58.682351 26850 ts_manager.cc:194] Registered new tserver with Master: 155d8694981a484db1bc03968535de49 (127.20.199.2:41977)
I20260925 14:53:58.682823 26850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:51893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:53:58.698527 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:41949
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:58.778821 27180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:58.779022 27180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:58.779044 27180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:58.780189 27180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:58.780231 27180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:58.780257 27180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:53:58.781716 27180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:58.781934 27180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:58.782157 27180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:58.787065 27185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:58.787177 27186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:58.787145 27188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:58.787375 27180 server_base.cc:1034] running on GCE node
I20260925 14:53:58.787650 27180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:58.787892 27180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:58.789077 27180 hybrid_clock.cc:648] HybridClock initialized: now 1790348038789067 us; error 33 us; skew 500 ppm
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.791734 27180 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:53:58.792135 27180 webserver.cc:460] Webserver started at http://127.20.199.3:35679/ using document root <none> and password file <none>
I20260925 14:53:58.792315 27180 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:58.792357 27180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:58.792447 27180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:58.793565 27180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "d83c48bf1b54401a9fea3e1bdd977b03"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "dcf37096718d709a97e5275d1dd3fa11"
server_key_iv: "d10e508cdbf5eb3551fff673ac0fa6e6"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.793898 27180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "d83c48bf1b54401a9fea3e1bdd977b03"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "dcf37096718d709a97e5275d1dd3fa11"
server_key_iv: "d10e508cdbf5eb3551fff673ac0fa6e6"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.795193 27180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:53:58.795969 27195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.796124 27180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:58.796209 27180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "d83c48bf1b54401a9fea3e1bdd977b03"
format_stamp: "Formatted at 2026-09-25 14:53:58 on dist-test-slave-8qt6"
server_key: "dcf37096718d709a97e5275d1dd3fa11"
server_key_iv: "d10e508cdbf5eb3551fff673ac0fa6e6"
server_key_version: "encryptionkey@0"
I20260925 14:53:58.796262 27180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:58.804729 27180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:58.806883 27180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:58.806990 27180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:58.807160 27180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:58.807410 27180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:58.807439 27180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.807459 27180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:58.807472 27180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:58.812781 27180 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:34991
I20260925 14:53:58.812868 27308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:34991 every 8 connection(s)
I20260925 14:53:58.813165 27180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:53:58.822618 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.813981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:41817 (local address 127.20.199.62:41949)
0925 14:53:58.814164 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.814167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.814322 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:53:58.815274 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.815275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.815289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.815339 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.815946 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.816452 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.817467 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.817674 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.817757 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.820035 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.820037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.820037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.820046 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.821369 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.821811 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.821811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.821812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.821864 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.822084 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.822084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.822084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.822189 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.822239 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.822462 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.822552 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260925 14:53:58.822748 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.813890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:41949 (local address 127.20.199.3:41817)
0925 14:53:58.814194 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:58.814238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:58.815159 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:58.815405 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:58.815408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:58.815657 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:58.815844 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:58.815849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.816541 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.816542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:58.817338 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:58.817340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.817765 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.817766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:58.817888 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.818387 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:58.818395 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:58.819910 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:58.821527 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:58.821532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:58.821534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:58.821746 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:58.821917 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:58.821918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:58.821918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:58.822010 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:58.822241 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:58.822245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:58.822354 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 14:53:58.822435 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:53:58.822551 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260925 14:53:58.823129 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27180
I20260925 14:53:58.823246 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:58.823220 27309 heartbeater.cc:347] Connected to a master server at 127.20.199.62:41949
I20260925 14:53:58.823313 27309 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:58.823459 21276 external_mini_cluster.cc:1428] Setting key f6d95abc5ba75ab0bdcf0d7737f9d03b
I20260925 14:53:58.823510 27309 heartbeater.cc:511] Master 127.20.199.62:41949 requested a full tablet report, sending...
I20260925 14:53:58.823947 26850 ts_manager.cc:194] Registered new tserver with Master: d83c48bf1b54401a9fea3e1bdd977b03 (127.20.199.3:34991)
I20260925 14:53:58.824331 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:58.824333 26850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:41817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/test-admin.keytab.
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.839963 21276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:53:58.841498 27316 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:58.840771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.20.199.1:44163)
0925 14:53:58.840919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.840923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.840934 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:58.840961 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.840963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.840976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.841050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.841359 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40284: BlockingRecv error: recv got EOF from 127.0.0.1:40284 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260925 14:53:58.841624 21276 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc)
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.842617 21276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.843504 21276 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:58.852147 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.844099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.20.199.1:44163)
0925 14:53:58.844147 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.844149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.844285 (+ 136us) server_negotiation.cc:407] Connection header received
0925 14:53:58.844313 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.844314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.844327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.844386 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.844748 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.845561 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.846270 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.846617 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.846678 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.848813 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.848821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.848821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.848829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.850952 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.851363 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.851367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.851367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.851437 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.851606 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.851607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.851608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.851759 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.851863 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.851923 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.851988 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:53:58.854367 27316 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:58.853568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.20.199.1:44163)
0925 14:53:58.853607 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.853609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.853641 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:53:58.853688 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.853689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.853699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.853753 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.854311 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40308: BlockingRecv error: recv got EOF from 127.0.0.1:40308 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:53:58.854427 21276 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc)
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.855362 21276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.856200 21276 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:58.863438 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.856742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.20.199.1:44163)
0925 14:53:58.856802 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.856804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.856870 (+ 66us) server_negotiation.cc:407] Connection header received
0925 14:53:58.856949 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.856950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.856962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.857025 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.857440 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.857922 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.858607 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.858822 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.858915 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.861034 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.861035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.861035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.861043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.862495 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.862805 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.862806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.862806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.862856 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.863007 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.863007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.863008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.863093 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.863183 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.863246 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.863305 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 14:53:58.864939 27316 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:58.864351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.20.199.1:44163)
0925 14:53:58.864407 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.864409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.864420 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:58.864493 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.864493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.864502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.864557 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.864892 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40320: BlockingRecv error: recv got EOF from 127.0.0.1:40320 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 14:53:58.865012 21276 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348014035135-21276-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.865980 21276 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:58.866874 21276 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[26805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:53:58.873775 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:58.867381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.20.199.1:44163)
0925 14:53:58.867440 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:58.867442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:58.867496 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:53:58.867602 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:58.867603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:58.867612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:58.867661 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:58.868016 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.868511 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.869189 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:58.869387 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:58.869488 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:58.871537 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:58.871538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:58.871538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:58.871545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:58.872644 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.872985 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.872987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.872988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.873045 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:58.873266 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:58.873267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:58.873267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:58.873362 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:58.873492 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:58.873606 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:53:58.873679 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:53:58.874418 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26912
I20260925 14:53:58.879624 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27046
I20260925 14:53:58.883865 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27180
I20260925 14:53:58.887920 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 26818
2026-09-25T14:53:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1860 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[27329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[27329](info): set up 2 sockets
Sep 25 14:53:58 dist-test-slave-8qt6 krb5kdc[27329](info): commencing operation
krb5kdc: starting...
W20260925 14:53:59.929733 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab.
Entry for principal oryx/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab.
I20260925 14:54:00.051888 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40765
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab
--principal=oryx/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:40765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:00.130934 27342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:00.131172 27342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:00.131207 27342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:00.132330 27342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:00.132400 27342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:00.132426 27342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:00.132447 27342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:00.132474 27342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:00.133941 27342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:40765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab
--principal=oryx/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:00.134238 27342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:00.134464 27342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:00.139299 27348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.139422 27342 server_base.cc:1034] running on GCE node
W20260925 14:54:00.139307 27347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.139374 27350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.139811 27342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:00.140086 27342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:00.141225 27342 hybrid_clock.cc:648] HybridClock initialized: now 1790348040141212 us; error 29 us; skew 500 ppm
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:00.143628 27342 init.cc:377] Logged in from keytab as oryx/127.20.199.62@KRBTEST.COM (short username oryx)
I20260925 14:54:00.144074 27342 webserver.cc:460] Webserver started at http://127.20.199.62:33495/ using document root <none> and password file <none>
I20260925 14:54:00.144302 27342 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.144346 27342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.144501 27342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:00.145568 27342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "684c9a750c1148949f3cd4cc17d297cb"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "9570cae5f4e4a103e8b7f56dbb794592"
server_key_iv: "18b46cc5200884a60e26249748fa532f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.145919 27342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "684c9a750c1148949f3cd4cc17d297cb"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "9570cae5f4e4a103e8b7f56dbb794592"
server_key_iv: "18b46cc5200884a60e26249748fa532f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.147177 27342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:54:00.147876 27357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.148041 27342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.148136 27342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "684c9a750c1148949f3cd4cc17d297cb"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "9570cae5f4e4a103e8b7f56dbb794592"
server_key_iv: "18b46cc5200884a60e26249748fa532f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.148222 27342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.165216 27342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:00.166914 27342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:00.167068 27342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:00.170958 27342 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:40765
I20260925 14:54:00.170992 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:40765 every 8 connection(s)
I20260925 14:54:00.171344 27342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:00.172259 27410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:00.174577 27410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Bootstrap starting.
I20260925 14:54:00.175104 27410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:00.175338 27410 log.cc:824] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:00.175944 27410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: No bootstrap required, opened a new log
I20260925 14:54:00.176534 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27342
I20260925 14:54:00.176641 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:54:00.176867 21276 external_mini_cluster.cc:1428] Setting key bf5ae0cfdece8b29c29ddf4791536fb8
I20260925 14:54:00.178082 27410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } }
I20260925 14:54:00.178259 27410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:00.178298 27410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684c9a750c1148949f3cd4cc17d297cb, State: Initialized, Role: FOLLOWER
I20260925 14:54:00.178439 27410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } }
I20260925 14:54:00.178524 27410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:00.178560 27410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:00.178608 27410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:00.179560 27410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } }
I20260925 14:54:00.179690 27410 leader_election.cc:302] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 684c9a750c1148949f3cd4cc17d297cb; no voters:
I20260925 14:54:00.179858 27410 leader_election.cc:288] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:00.179893 27415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:00.180011 27415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 LEADER]: Becoming Leader. State: Replica: 684c9a750c1148949f3cd4cc17d297cb, State: Running, Role: LEADER
I20260925 14:54:00.180120 27410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:00.180087 27415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } }
I20260925 14:54:00.180440 27416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } }
I20260925 14:54:00.180675 27416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:00.180598 27417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 684c9a750c1148949f3cd4cc17d297cb. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } }
I20260925 14:54:00.180936 27417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:00.181234 27424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:00.181766 27424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:00.183295 27424 catalog_manager.cc:1347] Generated new cluster ID: a9e4316ad58a45548f6d7685232580fe
I20260925 14:54:00.183352 27424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:00.188592 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.177381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.20.199.62:40765)
0925 14:54:00.177550 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.177570 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.177607 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:54:00.178062 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.178068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.178115 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.178230 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.179040 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.180108 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.181225 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.181861 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.182073 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.184733 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.184738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.184738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.184749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.186663 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.187063 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.187066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.187067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.187195 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.187404 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.187405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.187405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.187677 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.188210 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.188271 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.188407 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260925 14:54:00.199343 27424 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:00.199822 27424 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:00.205375 27424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Generated new TSK 0
I20260925 14:54:00.205566 27424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab.
Entry for principal oryx/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab.
I20260925 14:54:00.236680 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40765
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:00.317971 27436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:00.318208 27436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:00.318248 27436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:00.319465 27436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:00.319528 27436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:00.319573 27436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:00.321086 27436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:00.321341 27436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:00.321623 27436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:00.326686 27442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.326673 27444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.326674 27441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.327093 27436 server_base.cc:1034] running on GCE node
I20260925 14:54:00.327314 27436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:00.327548 27436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:00.328709 27436 hybrid_clock.cc:648] HybridClock initialized: now 1790348040328688 us; error 37 us; skew 500 ppm
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:00.331740 27436 init.cc:377] Logged in from keytab as oryx/127.20.199.1@KRBTEST.COM (short username oryx)
I20260925 14:54:00.332201 27436 webserver.cc:460] Webserver started at http://127.20.199.1:38423/ using document root <none> and password file <none>
I20260925 14:54:00.332397 27436 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.332437 27436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.332530 27436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:00.333604 27436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "ff97d18e016b4ca69b9e5f05af95ccf2"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "206161e9ebd43eb6f6e78ea4e4e89a4c"
server_key_iv: "6e33e18ddf48abcc90de78317a34a719"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.333915 27436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "ff97d18e016b4ca69b9e5f05af95ccf2"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "206161e9ebd43eb6f6e78ea4e4e89a4c"
server_key_iv: "6e33e18ddf48abcc90de78317a34a719"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.335187 27436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:00.335904 27451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.336077 27436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:00.336162 27436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "ff97d18e016b4ca69b9e5f05af95ccf2"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "206161e9ebd43eb6f6e78ea4e4e89a4c"
server_key_iv: "6e33e18ddf48abcc90de78317a34a719"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.336233 27436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.364982 27436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:00.367206 27436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:00.367364 27436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:00.367621 27436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:00.367952 27436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:00.368014 27436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.368058 27436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:00.368085 27436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.374039 27436 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:38101
I20260925 14:54:00.374094 27564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:38101 every 8 connection(s)
I20260925 14:54:00.374519 27436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:00.381785 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27436
I20260925 14:54:00.381876 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:00.382154 21276 external_mini_cluster.cc:1428] Setting key 0a4b4bc3c1fe149cdccda48ecec2b066
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:00.385986 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.376519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:50043 (local address 127.20.199.62:40765)
0925 14:54:00.376593 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.376595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.376609 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:54:00.377267 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.377268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.377286 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.377366 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.378374 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.378889 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.379937 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.380178 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.380235 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.382775 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.382776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.382777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.382785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.384419 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.384929 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.384930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.384930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.385007 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.385290 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.385291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.385291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.385442 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.385509 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.385801 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.385897 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:00.386144 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.376160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:50043)
0925 14:54:00.376340 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.376412 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.377155 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.377480 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.377486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.378069 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.378269 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.378273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.378971 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.378971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.379839 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.379841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.380249 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.380250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.380355 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.380989 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.381002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.382674 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.384505 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.384512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.384515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.384850 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.385073 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.385074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.385074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.385218 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.385499 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.385505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.385646 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.385759 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.385912 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260925 14:54:00.386617 27565 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40765
I20260925 14:54:00.386741 27565 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:00.386955 27565 heartbeater.cc:511] Master 127.20.199.62:40765 requested a full tablet report, sending...
I20260925 14:54:00.387465 27374 ts_manager.cc:194] Registered new tserver with Master: ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101)
I20260925 14:54:00.388123 27374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:50043
WARNING: no policy specified for oryx/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab.
Entry for principal oryx/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab.
I20260925 14:54:00.409194 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40765
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:00.490502 27570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:00.490718 27570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:00.490741 27570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:00.491886 27570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:00.491937 27570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:00.491964 27570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:00.493722 27570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:00.493955 27570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:00.494197 27570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:00.499104 27575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.499121 27578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.499173 27576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.499680 27570 server_base.cc:1034] running on GCE node
I20260925 14:54:00.499872 27570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:00.500129 27570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:00.501293 27570 hybrid_clock.cc:648] HybridClock initialized: now 1790348040501277 us; error 26 us; skew 500 ppm
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:00.504050 27570 init.cc:377] Logged in from keytab as oryx/127.20.199.2@KRBTEST.COM (short username oryx)
I20260925 14:54:00.504546 27570 webserver.cc:460] Webserver started at http://127.20.199.2:37165/ using document root <none> and password file <none>
I20260925 14:54:00.504770 27570 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.504839 27570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.504997 27570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:00.506139 27570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "634f787031284d6f8d70941f91850b6d"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "60e6034c28f48401ac4a8a0221e9c878"
server_key_iv: "139a0f447f2bf1d6d600fbdd182f990e"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.506601 27570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "634f787031284d6f8d70941f91850b6d"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "60e6034c28f48401ac4a8a0221e9c878"
server_key_iv: "139a0f447f2bf1d6d600fbdd182f990e"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.507958 27570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:00.508735 27585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.508924 27570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:00.509027 27570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "634f787031284d6f8d70941f91850b6d"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "60e6034c28f48401ac4a8a0221e9c878"
server_key_iv: "139a0f447f2bf1d6d600fbdd182f990e"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.509117 27570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.529364 27570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:00.531275 27570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:00.531442 27570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:00.531692 27570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:00.532019 27570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:00.532074 27570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.532109 27570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:00.532135 27570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.538702 27570 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44423
I20260925 14:54:00.538739 27698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44423 every 8 connection(s)
I20260925 14:54:00.539132 27570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:00.544229 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27570
I20260925 14:54:00.544318 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:00.544528 21276 external_mini_cluster.cc:1428] Setting key 4acc296602deae2b8660a0280bc3e252
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:00.549314 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.539873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:33481 (local address 127.20.199.62:40765)
0925 14:54:00.539940 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.539942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.540190 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:54:00.540653 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.540653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.540663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.540728 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.541753 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.542286 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.543257 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.543454 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.543522 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.546203 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.546204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.546204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.546211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.547902 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.548293 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.548294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.548294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.548361 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.548597 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.548598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.548598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.548750 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.548820 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.549061 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.549160 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:54:00.549391 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.539862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:33481)
0925 14:54:00.540046 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.540097 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.540575 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.540783 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.540786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.541361 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.541632 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.541638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.542368 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.542369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.543176 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.543178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.543538 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.543539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.543639 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.544201 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.544209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.546109 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.547984 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.547990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.547992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.548218 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.548430 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.548430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.548431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.548527 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.548814 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.548817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.548931 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.549013 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.549204 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260925 14:54:00.549901 27699 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40765
I20260925 14:54:00.550031 27699 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:00.550259 27699 heartbeater.cc:511] Master 127.20.199.62:40765 requested a full tablet report, sending...
I20260925 14:54:00.550688 27374 ts_manager.cc:194] Registered new tserver with Master: 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423)
I20260925 14:54:00.551244 27374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:33481
WARNING: no policy specified for oryx/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab.
Entry for principal oryx/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab.
I20260925 14:54:00.580772 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40765
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:00.661749 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:00.661955 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:00.661978 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:00.663141 27704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:00.663182 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:00.663208 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:00.664629 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:00.664868 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:00.665107 27704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:00.670012 27712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.670030 27709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.670259 27704 server_base.cc:1034] running on GCE node
W20260925 14:54:00.670030 27710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.670548 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:00.670785 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:00.671942 27704 hybrid_clock.cc:648] HybridClock initialized: now 1790348040671934 us; error 37 us; skew 500 ppm
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:00.674533 27704 init.cc:377] Logged in from keytab as oryx/127.20.199.3@KRBTEST.COM (short username oryx)
I20260925 14:54:00.674914 27704 webserver.cc:460] Webserver started at http://127.20.199.3:46203/ using document root <none> and password file <none>
I20260925 14:54:00.675189 27704 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.675261 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.675413 27704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:00.676494 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "edec8472aec940b2995695543883ba11"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "5120acf814054623d0659a43584e8522"
server_key_iv: "eeca7f6455c86d5bba020cf21ec23288"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.676838 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "edec8472aec940b2995695543883ba11"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "5120acf814054623d0659a43584e8522"
server_key_iv: "eeca7f6455c86d5bba020cf21ec23288"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.678222 27704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:00.678946 27719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.679116 27704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:00.679214 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "edec8472aec940b2995695543883ba11"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "5120acf814054623d0659a43584e8522"
server_key_iv: "eeca7f6455c86d5bba020cf21ec23288"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.679291 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.687147 27704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:00.689113 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:00.689270 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:00.689543 27704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:00.689845 27704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:00.689898 27704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.689939 27704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:00.689972 27704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.695626 27704 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:44885
I20260925 14:54:00.695694 27832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:44885 every 8 connection(s)
I20260925 14:54:00.696000 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:00.704814 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.696746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:41775 (local address 127.20.199.62:40765)
0925 14:54:00.696821 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.696823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.696984 (+ 161us) server_negotiation.cc:407] Connection header received
0925 14:54:00.697531 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.697532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.697543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.697594 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.698457 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.698943 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.699858 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.700031 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.700084 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.702206 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.702207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.702207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.702213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.703491 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.703944 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.703944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.703945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.703998 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.704264 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.704264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.704264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.704367 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.704418 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.704648 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.704707 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:54:00.704941 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.696700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:41775)
0925 14:54:00.696865 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.696900 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.697383 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.697650 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.697653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.698194 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.698385 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.698389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.699029 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.699029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.699790 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.699792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.700098 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.700098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.700191 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.700720 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.700728 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.702130 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.703583 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.703589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.703591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.703836 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.704088 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.704089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.704089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.704199 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.704427 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.704431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.704539 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.704619 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.704737 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260925 14:54:00.705554 27833 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40765
I20260925 14:54:00.705681 27833 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:00.705654 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27704
I20260925 14:54:00.705745 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:00.705874 27833 heartbeater.cc:511] Master 127.20.199.62:40765 requested a full tablet report, sending...
I20260925 14:54:00.706005 21276 external_mini_cluster.cc:1428] Setting key 7b0a86d23e2f6c09fa4fb0697264af08
I20260925 14:54:00.706326 27374 ts_manager.cc:194] Registered new tserver with Master: edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:00.706807 27374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:41775
I20260925 14:54:00.706833 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.199.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.61.keytab.
Entry for principal oryx/127.20.199.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.61.keytab.
I20260925 14:54:00.731933 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:42661
--webserver_interface=127.20.199.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.61.keytab
--principal=oryx/127.20.199.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:40765,127.20.199.61:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:00.812397 27838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:00.812597 27838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:00.812618 27838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:00.813776 27838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:00.813817 27838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:00.813831 27838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:00.813843 27838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:00.813855 27838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:00.815299 27838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:40765,127.20.199.61:42661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.61.keytab
--principal=oryx/127.20.199.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.199.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
W20260925 14:54:00.815393 27838 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.199.62:40765,127.20.199.61:42661'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:54:00.815563 27838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:00.815799 27838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:00.820732 27843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.820751 27846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.820737 27844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.820966 27838 server_base.cc:1034] running on GCE node
I20260925 14:54:00.821219 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:00.821542 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:00.822712 27838 hybrid_clock.cc:648] HybridClock initialized: now 1790348040822707 us; error 37 us; skew 500 ppm
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:00.825217 27838 init.cc:377] Logged in from keytab as oryx/127.20.199.61@KRBTEST.COM (short username oryx)
I20260925 14:54:00.825734 27838 webserver.cc:460] Webserver started at http://127.20.199.61:44521/ using document root <none> and password file <none>
I20260925 14:54:00.825948 27838 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.825992 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.826108 27838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:00.827234 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/instance:
uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "bf917207aa5b452b336022b5011c7d21"
server_key_iv: "bd6a89c89e4f8b053f79b0054c0abe1f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.827618 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal/instance:
uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "bf917207aa5b452b336022b5011c7d21"
server_key_iv: "bd6a89c89e4f8b053f79b0054c0abe1f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.829036 27838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:00.829881 27853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.830087 27838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:00.830199 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "bf917207aa5b452b336022b5011c7d21"
server_key_iv: "bd6a89c89e4f8b053f79b0054c0abe1f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.830286 27838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.857622 27838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:00.860159 27838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:00.860340 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:00.864420 27838 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.61:42661
I20260925 14:54:00.864455 27905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.61:42661 every 8 connection(s)
I20260925 14:54:00.864822 27838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
I20260925 14:54:00.865850 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:00.866797 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 27838
I20260925 14:54:00.867748 27906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.61@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
I20260925 14:54:00.877547 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.868773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.20.199.62:40765)
0925 14:54:00.868848 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.868851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.868975 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:54:00.869012 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.869013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.869026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.869103 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.869656 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.870425 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.871303 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.871495 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.871591 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.874030 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.874031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.874032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.874048 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.875730 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.876256 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.876258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.876258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.876328 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.876620 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.876621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.876621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.876850 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.876981 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.877314 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.877404 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:00.877678 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.868691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:52038)
0925 14:54:00.868866 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.868910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.868969 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.869129 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.869131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.869357 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.869565 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.869570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.870518 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.870519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.871211 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.871213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.871612 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.871613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.871757 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.872258 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.872268 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.873934 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.875809 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.875814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.875816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.876163 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.876407 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.876408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.876408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.876549 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.876970 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.876977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.877152 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.877264 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.877442 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":15,"thread_start_us":104,"threads_started":1}
I20260925 14:54:00.877678 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.867466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.20.199.61:42661)
0925 14:54:00.867657 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.867667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.867703 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:54:00.868277 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.868284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.868376 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.868536 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.869422 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.870697 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.871504 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.871845 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.871942 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.874484 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.874491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.874491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.874498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.876404 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.876821 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.876823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.876823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.876868 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.877089 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.877089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.877090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.877241 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.877378 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.877524 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.877596 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260925 14:54:00.878566 27906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.61@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
I20260925 14:54:00.884883 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.878801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:33576)
0925 14:54:00.878852 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.878865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.878911 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.879020 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.879021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.879137 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.879171 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.879172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.879792 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.879794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.880340 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.880341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.880750 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.880752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.880866 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.881248 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.881253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.882555 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.884061 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.884062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.884062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.884231 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.884386 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.884386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.884386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.884443 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.884633 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.884634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.884666 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.884716 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.884796 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:54:00.884929 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.878854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.20.199.61:42661)
0925 14:54:00.878900 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.878902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.878910 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:00.878967 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.878968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.878975 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.879016 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.879223 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.879667 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.880422 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.880661 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.880737 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.882620 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.882621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.882621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.882627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.883988 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.884292 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.884292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.884293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.884333 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.884492 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.884492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.884493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.884583 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.884626 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.884736 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.884837 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":29}
I20260925 14:54:00.886440 27906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Bootstrap starting.
I20260925 14:54:00.887276 27906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:00.887560 27906 log.cc:824] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:00.888293 27906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: No bootstrap required, opened a new log
I20260925 14:54:00.889760 27906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } }
I20260925 14:54:00.889889 27906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:00.889911 27906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3d21a05a5941289d00c3aa6d0a28e9, State: Initialized, Role: FOLLOWER
I20260925 14:54:00.890031 27906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } }
I20260925 14:54:00.890290 27912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } } }
I20260925 14:54:00.890331 27906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:00.890372 27912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:54:00.892014 27923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:00.892071 27923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:00.893442 27838 master_runner.cc:428] Detected that this master 3f3d21a05a5941289d00c3aa6d0a28e9 is joining an existing cluster
I20260925 14:54:00.893493 27838 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.199.61:42661
I20260925 14:54:00.901742 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.894723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:33592)
0925 14:54:00.894847 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.894858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.894933 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.895176 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.895177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.895388 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.895475 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.895477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.896281 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.896282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.897054 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.897055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.897573 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.897574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.897635 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.898146 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.898150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.898484 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.900583 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.900584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.900584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.900762 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.900990 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.900991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.900992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.901063 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.901327 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.901328 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.901376 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.901477 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.901594 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260925 14:54:00.901952 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.894577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.20.199.62:40765)
0925 14:54:00.894630 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.894633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.894961 (+ 328us) server_negotiation.cc:407] Connection header received
0925 14:54:00.895065 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.895066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.895080 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.895135 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.895769 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.896534 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.897455 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.897741 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.897819 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.899077 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.899078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.899078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.899086 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.900713 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.901123 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.901124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.901124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.901182 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.901376 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.901376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.901377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.901528 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.901574 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.901780 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.901860 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:54:00.901755 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.894786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.20.199.61:42661)
0925 14:54:00.894825 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.894827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.894931 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:54:00.895000 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.895001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.895011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.895101 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.895568 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.896186 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.897141 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.897482 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.897555 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.898591 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.898592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.898593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.898600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.900475 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.900851 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.900859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.900860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.900921 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.901129 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.901130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.901130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.901246 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.901334 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.901500 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.901663 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":20}
I20260925 14:54:00.902287 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.894511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:52054)
0925 14:54:00.894847 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.894901 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.894987 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.895412 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.895413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.895633 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.895679 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.895681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.896624 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.896625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.897336 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.897337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.898136 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.898137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.898204 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.898616 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.898621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.898986 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.900808 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.900809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.900809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.901019 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.901253 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.901254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.901254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.901309 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.901623 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.901624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.901666 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.902084 (+ 418us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.902182 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":86,"threads_started":1}
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.62@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
I20260925 14:54:00.912076 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.904562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.20.199.61:42661)
0925 14:54:00.904606 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.904608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.904777 (+ 169us) server_negotiation.cc:407] Connection header received
0925 14:54:00.904861 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.904862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.904873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.904929 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.905521 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.905979 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.907085 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.907263 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.907324 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.909481 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.909482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.909483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.909490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.910825 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.911232 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.911233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.911233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.911295 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.911551 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.911552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.911552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.911662 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.911735 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.911911 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.911988 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:54:00.912109 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.904484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:33602)
0925 14:54:00.904657 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.904707 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.904785 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.904935 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.904938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.905308 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.905377 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.905379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.906065 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.906066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.906990 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.906993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.907345 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.907346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.907428 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.907866 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.907871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.909355 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.910925 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.910933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.910935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.911143 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.911418 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.911419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.911419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.911486 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.911729 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.911736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.911787 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.911886 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.912011 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260925 14:54:00.912654 27374 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.199.61:42661: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
member_type: NON_VOTER
last_known_addr {
host: "127.20.199.61"
port: 42661
}
attrs {
promote: true
}
}
dest_uuid: "684c9a750c1148949f3cd4cc17d297cb"
cas_config_opid_index: -1
I20260925 14:54:00.912974 27374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } }
I20260925 14:54:00.913443 27933 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } }
I20260925 14:54:00.913796 27880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:00.914546 27933 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.199.61:42661
I20260925 14:54:00.914746 27880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Refusing update from remote peer 684c9a750c1148949f3cd4cc17d297cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:54:00.914783 27838 master.cc:537] Master@127.20.199.61:42661 shutting down...
I20260925 14:54:00.914868 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } }
I20260925 14:54:00.914945 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:00.915041 27933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:54:00.915632 27361 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 14:54:00.916714 27361 consensus_peers.cc:487] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb -> Peer 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61:42661): Couldn't send request to peer 3f3d21a05a5941289d00c3aa6d0a28e9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:54:00.917596 27838 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:00.917682 27838 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:00.917734 27838 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: stopping tablet replica
I20260925 14:54:00.930404 27838 master.cc:552] Master@127.20.199.61:42661 shutdown complete.
I20260925 14:54:00.930471 27838 master_runner.cc:305] Clearing existing system tablet
I20260925 14:54:00.930612 27838 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.930660 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.931365 27838 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260925 14:54:00.931876 27937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.932029 27838 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.932101 27838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:00.932132 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "bf917207aa5b452b336022b5011c7d21"
server_key_iv: "bd6a89c89e4f8b053f79b0054c0abe1f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.933203 27838 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:00.934562 27838 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:54:00.934649 27838 log.cc:1196] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:54:00.934845 27838 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Deleting consensus metadata
I20260925 14:54:00.935020 27838 master_runner.cc:315] Copying system tablet from 127.20.199.62:40765
I20260925 14:54:00.935613 27838 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.199.62:40765
I20260925 14:54:00.945773 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.936007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.20.199.62:40765)
0925 14:54:00.936055 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.936057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.936092 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:54:00.936170 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.936171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.936180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.936262 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.936616 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.937093 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.938188 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.938390 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.938454 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.942947 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.942949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.942950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.942958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.944354 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.944763 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.944765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.944765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.944854 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.945168 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.945170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.945170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.945354 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.945463 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.945633 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.945709 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:00.945824 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.935906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:52064)
0925 14:54:00.936024 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:00.936035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:00.936108 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:00.936269 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:00.936270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:00.936475 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:00.936546 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.936548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.937168 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.937169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.938087 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:00.938089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.938472 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.938473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:00.938538 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.942447 (+ 3909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:00.942453 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:00.942851 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:00.944441 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.944442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.944443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.944668 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.944960 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:00.944961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:00.944962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:00.945059 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:00.945463 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:00.945464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:00.945531 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 14:54:00.945600 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:54:00.945705 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":78,"threads_started":1}
I20260925 14:54:00.946163 27394 tablet_copy_service.cc:140] P 684c9a750c1148949f3cd4cc17d297cb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f3d21a05a5941289d00c3aa6d0a28e9 ({username='oryx', principal='oryx/127.20.199.61@KRBTEST.COM'} at 127.0.0.1:52064)
I20260925 14:54:00.946273 27394 tablet_copy_service.cc:161] P 684c9a750c1148949f3cd4cc17d297cb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f3d21a05a5941289d00c3aa6d0a28e9 at {username='oryx', principal='oryx/127.20.199.61@KRBTEST.COM'} at 127.0.0.1:52064: session id = 3f3d21a05a5941289d00c3aa6d0a28e9-00000000000000000000000000000000
I20260925 14:54:00.947758 27394 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:54:00.948999 27838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:54:00.950098 27838 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: tablet copy: Starting download of 0 data blocks...
I20260925 14:54:00.950314 27838 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: tablet copy: Starting download of 1 WAL segments...
I20260925 14:54:00.951402 27838 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:54:00.952273 27394 tablet_copy_service.cc:342] P 684c9a750c1148949f3cd4cc17d297cb: Request end of tablet copy session 3f3d21a05a5941289d00c3aa6d0a28e9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.199.61@KRBTEST.COM'} at 127.0.0.1:52064
I20260925 14:54:00.952376 27394 tablet_copy_service.cc:433] P 684c9a750c1148949f3cd4cc17d297cb: ending tablet copy session 3f3d21a05a5941289d00c3aa6d0a28e9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f3d21a05a5941289d00c3aa6d0a28e9
I20260925 14:54:00.955078 27838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:00.956624 27945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.956779 27838 server_base.cc:1034] running on GCE node
W20260925 14:54:00.956892 27947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:00.957065 27944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:00.957231 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:00.957393 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:00.958590 27838 hybrid_clock.cc:648] HybridClock initialized: now 1790348040958575 us; error 31 us; skew 500 ppm
Sep 25 14:54:00 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:00.959929 27838 init.cc:377] Logged in from keytab as oryx/127.20.199.61@KRBTEST.COM (short username oryx)
I20260925 14:54:00.960217 27838 webserver.cc:460] Webserver started at http://127.20.199.61:44181/ using document root <none> and password file <none>
I20260925 14:54:00.960364 27838 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:00.960394 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:00.961077 27838 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:00.961668 27954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:00.961786 27838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:00.961833 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "bf917207aa5b452b336022b5011c7d21"
server_key_iv: "bd6a89c89e4f8b053f79b0054c0abe1f"
server_key_version: "encryptionkey@0"
I20260925 14:54:00.961892 27838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:00.974452 27838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:00.976081 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:00.979183 27838 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.61:42661
I20260925 14:54:00.979421 28006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.61:42661 every 8 connection(s)
I20260925 14:54:00.979744 27838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
I20260925 14:54:00.980528 28008 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:54:00.981094 28008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Bootstrap starting.
I20260925 14:54:00.983994 28008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:54:00.984223 28008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: Bootstrap complete.
I20260925 14:54:00.984422 28008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } }
I20260925 14:54:00.984514 28008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f3d21a05a5941289d00c3aa6d0a28e9, State: Initialized, Role: LEARNER
I20260925 14:54:00.984570 28008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } }
I20260925 14:54:00.984755 28008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:00.984757 28011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } }
I20260925 14:54:00.985268 28011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:54:00.986683 28022 catalog_manager.cc:1259] Loaded cluster ID: a9e4316ad58a45548f6d7685232580fe
I20260925 14:54:00.986770 28022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: loading cluster ID for follower catalog manager: success
I20260925 14:54:00.987699 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:00.979527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.20.199.61:42661)
0925 14:54:00.979669 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:00.979671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:00.979680 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:00.979708 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:00.979708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:00.979719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:00.980670 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:00.980708 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.981739 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.982801 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:00.983103 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:00.983182 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:00.984596 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:00.984598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:00.984598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:00.984605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:00.986461 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.986923 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.986924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.986925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.986989 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:00.987188 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:00.987189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:00.987189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:00.987301 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:00.987404 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:00.987509 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:54:00.987588 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260925 14:54:00.987911 28022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: acquiring CA information for follower catalog manager: success
I20260925 14:54:00.988219 28022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for oryx/127.20.199.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.60.keytab.
Entry for principal oryx/127.20.199.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.60.keytab.
I20260925 14:54:01.012425 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:41853
--webserver_interface=127.20.199.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.60.keytab
--principal=oryx/127.20.199.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:01.093709 28025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:01.093942 28025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:01.093978 28025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:01.095224 28025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:01.095297 28025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:01.095322 28025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:01.095353 28025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:01.095392 28025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:01.097110 28025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.60.keytab
--principal=oryx/127.20.199.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:41853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.199.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:01.097448 28025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:01.097693 28025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:01.102685 28033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.102685 28030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.102705 28031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.102934 28025 server_base.cc:1034] running on GCE node
I20260925 14:54:01.103178 28025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:01.103421 28025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:01.104557 28025 hybrid_clock.cc:648] HybridClock initialized: now 1790348041104549 us; error 27 us; skew 500 ppm
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:01.107499 28025 init.cc:377] Logged in from keytab as oryx/127.20.199.60@KRBTEST.COM (short username oryx)
I20260925 14:54:01.107956 28025 webserver.cc:460] Webserver started at http://127.20.199.60:38335/ using document root <none> and password file <none>
I20260925 14:54:01.108186 28025 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:01.108255 28025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:01.108423 28025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:01.109593 28025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/instance:
uuid: "bb753be525354422a7b3d2a3d659bcbc"
format_stamp: "Formatted at 2026-09-25 14:54:01 on dist-test-slave-8qt6"
server_key: "eba26cb372e1e1517bc44babdcb7becb"
server_key_iv: "f8f77e9ecbf1527f1740ea32494c5aaa"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.109965 28025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal/instance:
uuid: "bb753be525354422a7b3d2a3d659bcbc"
format_stamp: "Formatted at 2026-09-25 14:54:01 on dist-test-slave-8qt6"
server_key: "eba26cb372e1e1517bc44babdcb7becb"
server_key_iv: "f8f77e9ecbf1527f1740ea32494c5aaa"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.111328 28025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:01.112201 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.112367 28025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:01.112473 28025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
uuid: "bb753be525354422a7b3d2a3d659bcbc"
format_stamp: "Formatted at 2026-09-25 14:54:01 on dist-test-slave-8qt6"
server_key: "eba26cb372e1e1517bc44babdcb7becb"
server_key_iv: "f8f77e9ecbf1527f1740ea32494c5aaa"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.112556 28025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:01.128099 28025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:01.130487 28025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:01.130674 28025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:01.134498 28025 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.60:41853
I20260925 14:54:01.134532 28092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.60:41853 every 8 connection(s)
I20260925 14:54:01.134878 28025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
I20260925 14:54:01.135882 28093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:01.137363 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 28025
I20260925 14:54:01.137555 28093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.60@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.199.60@KRBTEST.COM
I20260925 14:54:01.146818 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.138431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.20.199.62:40765)
0925 14:54:01.138479 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.138481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.139184 (+ 703us) server_negotiation.cc:407] Connection header received
0925 14:54:01.139221 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.139222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.139235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.139308 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.139873 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.140418 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.141287 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.141567 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.141658 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.143722 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.143723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.143723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.143729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.145498 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.145877 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.145878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.145879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.145952 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.146175 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.146176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.146177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.146305 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.146379 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.146645 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.146731 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:01.147017 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.138254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:52076)
0925 14:54:01.139081 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.139117 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.139177 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.139343 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.139345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.139622 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.139773 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.139778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.140529 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.140530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.141197 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.141199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.141685 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.141687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.141758 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.142174 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.142182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.143641 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.145582 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.145587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.145589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.145805 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.146019 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.146020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.146020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.146101 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.146384 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.146390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.146509 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.146608 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.146752 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":86,"threads_started":1}
I20260925 14:54:01.147541 28093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:01.147658 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.137999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.20.199.60:41853)
0925 14:54:01.138174 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.138183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.138217 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:54:01.138702 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.138710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.138767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.138912 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.139636 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.140529 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.141258 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.141634 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.141757 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.144320 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.144327 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.144327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.144334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.146072 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.146401 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.146404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.146405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.146457 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.146680 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.146681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.146681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.146813 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.147089 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.147492 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.147567 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.60@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
I20260925 14:54:01.154081 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.147845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.20.199.61:42661)
0925 14:54:01.147951 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.147954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.148181 (+ 227us) server_negotiation.cc:407] Connection header received
0925 14:54:01.148248 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.148249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.148263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.148307 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.148573 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.149053 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.149788 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.150015 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.150088 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.151893 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.151894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.151895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.151903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.153028 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.153347 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.153348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.153349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.153452 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.153676 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.153676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.153677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.153789 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.153842 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.153948 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.154020 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:54:01.154081 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.147787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:33624)
0925 14:54:01.147832 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.148129 (+ 297us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.148191 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.148317 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.148318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.148455 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.148489 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.148491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.149130 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.149131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.149700 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.149701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.150084 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.150085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.150147 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.150562 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.150567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.151825 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.153110 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.153111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.153111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.153275 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.153533 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.153534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.153534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.153614 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.153846 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.153846 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.153880 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.153931 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.154017 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:54:01.154469 28093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.60@KRBTEST.COM for oryx/127.20.199.60@KRBTEST.COM
I20260925 14:54:01.160003 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.154759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:33446)
0925 14:54:01.154808 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.154819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.154867 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.154962 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.154962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.155081 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.155109 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.155110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.155669 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.155669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.156123 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.156124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.156429 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.156430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.156463 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.156758 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.156762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.157940 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.159192 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.159193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.159193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.159350 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.159524 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.159524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.159525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.159576 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.159767 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.159768 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.159799 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.159848 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.159923 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:54:01.160086 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.154799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.20.199.60:41853)
0925 14:54:01.154840 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.154841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.154847 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:54:01.154904 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.154904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.154912 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.154953 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.155164 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.155609 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.156184 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.156364 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.156422 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.158023 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.158024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.158024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.158030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.159122 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.159423 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.159424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.159424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.159468 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.159629 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.159629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.159630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.159714 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.159760 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.159862 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.159964 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":19}
I20260925 14:54:01.161304 28093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Bootstrap starting.
I20260925 14:54:01.162263 28093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:01.162537 28093 log.cc:824] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:01.163260 28093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: No bootstrap required, opened a new log
I20260925 14:54:01.164669 28093 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } }
I20260925 14:54:01.164808 28093 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:01.164855 28093 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb753be525354422a7b3d2a3d659bcbc, State: Initialized, Role: FOLLOWER
I20260925 14:54:01.164994 28093 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } }
I20260925 14:54:01.165277 28093 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:01.165251 28098 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } } }
I20260925 14:54:01.165364 28098 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:54:01.166888 28109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:01.166949 28109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:01.169085 28025 master_runner.cc:428] Detected that this master bb753be525354422a7b3d2a3d659bcbc is joining an existing cluster
I20260925 14:54:01.169134 28025 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.199.60:41853
I20260925 14:54:01.199098 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27436
I20260925 14:54:01.202375 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.170198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:33450)
0925 14:54:01.170370 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.170381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.170446 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.170612 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.170612 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.170843 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.170919 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.170925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.171781 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.171781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.172628 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.172629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.173092 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.173093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.173161 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.173635 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.173640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.174463 (+ 823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.176113 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.176113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.176114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.176317 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.201634 (+ 25317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.201635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.201636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.201707 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.202007 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.202008 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.202051 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.202118 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.202227 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260925 14:54:01.202662 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.170234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.20.199.60:41853)
0925 14:54:01.170275 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.170277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.170453 (+ 176us) server_negotiation.cc:407] Connection header received
0925 14:54:01.170525 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.170526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.170536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.170608 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.171009 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.171707 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.172700 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.172965 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.173031 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.174558 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.174559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.174560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.174565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.176033 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.176591 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.176592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.176592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.201545 (+ 24953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.201773 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.201774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.201774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.201911 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.201972 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.202495 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.202571 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:54:01.204957 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:38101
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:40765
--webserver_port=38423
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:01.205498 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.170256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.20.199.62:40765)
0925 14:54:01.170300 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.170301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.171092 (+ 791us) server_negotiation.cc:407] Connection header received
0925 14:54:01.171179 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.171180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.171190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.171220 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.171724 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.172471 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.173533 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.173835 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.173883 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.201826 (+ 27943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.201827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.201828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.201837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.203651 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.204032 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.204033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.204034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.204090 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.204297 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.204298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.204298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.204430 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.204476 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.204993 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.205082 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:01.205859 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.170530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:52092)
0925 14:54:01.171043 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.171051 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.171114 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.171274 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.171274 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.171466 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.171528 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.171529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.172573 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.172574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.173420 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.173422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.173959 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.173960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.174018 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.174434 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.174439 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.201718 (+ 27279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.203744 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.203746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.203746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.203950 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.204164 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.204164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.204165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.204222 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.204809 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.204810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.204862 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.205581 (+ 719us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.205703 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":78,"threads_started":1}
I20260925 14:54:01.207367 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.170771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.20.199.61:42661)
0925 14:54:01.173497 (+ 2726us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.173499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.173508 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:01.173529 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.173530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.173540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.173598 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.174075 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.174795 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.175781 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.176129 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.176194 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.204331 (+ 28137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.204333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.204334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.204343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.206022 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.206503 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.206504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.206505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.206560 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.206735 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.206737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.206737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.206875 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.206947 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.207188 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.207277 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2704}
I20260925 14:54:01.207681 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.170721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:33632)
0925 14:54:01.171616 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.171624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.171656 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.173649 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.173650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.173927 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.173982 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.173983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.174891 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.174892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.175690 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.175692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.176191 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.176192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.176256 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.177031 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.177036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.204226 (+ 27190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.206202 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.206203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.206203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.206413 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.206627 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.206628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.206628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.206678 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.206951 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.206951 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.206999 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.207480 (+ 481us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.207579 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":117,"threads_started":1}
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348040, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.62@KRBTEST.COM for oryx/127.20.199.60@KRBTEST.COM
I20260925 14:54:01.216785 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.208716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.20.199.60:41853)
0925 14:54:01.208757 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.208760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.208768 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:01.208824 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.208825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.208836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.208890 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.209192 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.209951 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.210973 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.211340 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.211413 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.213753 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.213755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.213755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.213762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.215443 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.215937 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.215938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.215938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.215995 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.216253 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.216253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.216254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.216376 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.216443 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.216619 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.216692 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:01.216845 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.208663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:33464)
0925 14:54:01.208706 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.208717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.208762 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.208913 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.208914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.209070 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.209115 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.209116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.210046 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.210047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.210875 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.210877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.211432 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.211433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.211504 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.211939 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.211944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.213665 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.215595 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.215596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.215597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.215836 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.216070 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.216071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.216071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.216185 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.216446 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.216447 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.216502 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.216586 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.216698 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:54:01.217760 27374 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.199.60:41853: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc"
member_type: NON_VOTER
last_known_addr {
host: "127.20.199.60"
port: 41853
}
attrs {
promote: true
}
}
dest_uuid: "684c9a750c1148949f3cd4cc17d297cb"
cas_config_opid_index: 5
I20260925 14:54:01.218012 27374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } }
I20260925 14:54:01.218343 27932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb753be525354422a7b3d2a3d659bcbc (127.20.199.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.219291 27932 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.199.60:41853
I20260925 14:54:01.219357 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.219421 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:01.219946 28067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:01.220158 28025 master.cc:537] Master@127.20.199.60:41853 shutting down...
I20260925 14:54:01.220952 28067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Refusing update from remote peer 684c9a750c1148949f3cd4cc17d297cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:54:01.221271 27933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:54:01.222409 27358 consensus_peers.cc:487] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb -> Peer bb753be525354422a7b3d2a3d659bcbc (127.20.199.60:41853): Couldn't send request to peer bb753be525354422a7b3d2a3d659bcbc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:54:01.223668 28025 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:01.223753 28025 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:01.223796 28025 tablet_replica.cc:330] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: stopping tablet replica
I20260925 14:54:01.225612 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.219602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:33638)
0925 14:54:01.219860 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.219870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.219930 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.222218 (+ 2288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.222220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.222222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:01.222266 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.222267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.223214 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.223215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.224422 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.224425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.225300 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.225301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.225375 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.225417 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.225474 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.225519 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260925 14:54:01.226812 28025 master.cc:552] Master@127.20.199.60:41853 shutdown complete.
I20260925 14:54:01.226857 28025 master_runner.cc:305] Clearing existing system tablet
I20260925 14:54:01.226985 28025 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:01.227039 28025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:01.227694 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.219667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.20.199.61:42661)
0925 14:54:01.219708 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.219710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.219982 (+ 272us) server_negotiation.cc:407] Connection header received
0925 14:54:01.220022 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.220022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.220045 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.220098 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:01.222351 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.223126 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.224519 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.225125 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.225191 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.225262 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.227585 (+ 2323us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.227630 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:01.227885 27981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 LEARNER]: Refusing update from remote peer 684c9a750c1148949f3cd4cc17d297cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:54:01.228068 28025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:01.228060 27933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:54:01.228664 27981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb753be525354422a7b3d2a3d659bcbc (127.20.199.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.229628 28120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.229743 28025 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:01.229784 28025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:01.229803 28025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
uuid: "bb753be525354422a7b3d2a3d659bcbc"
format_stamp: "Formatted at 2026-09-25 14:54:01 on dist-test-slave-8qt6"
server_key: "eba26cb372e1e1517bc44babdcb7becb"
server_key_iv: "f8f77e9ecbf1527f1740ea32494c5aaa"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.229820 28011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 684c9a750c1148949f3cd4cc17d297cb. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.229907 28011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:54:01.229941 28011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: NON_VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: true } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.229980 28011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:54:01.230221 27933 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: attempting to promote NON_VOTER 3f3d21a05a5941289d00c3aa6d0a28e9 to VOTER
I20260925 14:54:01.230404 27933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } }
I20260925 14:54:01.230602 28025 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:01.231029 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.230921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:33478)
0925 14:54:01.230961 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.230999 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 14:54:01.231281 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.231218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:33480)
0925 14:54:01.231245 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.231264 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 14:54:01.231390 27358 consensus_peers.cc:487] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb -> Peer bb753be525354422a7b3d2a3d659bcbc (127.20.199.60:41853): Couldn't send request to peer bb753be525354422a7b3d2a3d659bcbc. Status: Network error: Client connection negotiation failed: client connection to 127.20.199.60:41853: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:54:01.231572 27981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 LEARNER]: Refusing update from remote peer 684c9a750c1148949f3cd4cc17d297cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:54:01.231727 27932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:54:01.232158 28025 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:54:01.232223 28025 log.cc:1196] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:54:01.232431 28025 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Deleting consensus metadata
I20260925 14:54:01.232539 28025 master_runner.cc:315] Copying system tablet from 127.20.199.62:40765
I20260925 14:54:01.232563 27932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.233115 28025 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: tablet copy: Beginning tablet copy session from remote peer at address 127.20.199.62:40765
I20260925 14:54:01.233388 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.233696 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:01.233749 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.233793 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:01.233914 27981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.234741 28011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.234822 28011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:01.242252 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.235268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.20.199.62:40765)
0925 14:54:01.235306 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.235308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.235588 (+ 280us) server_negotiation.cc:407] Connection header received
0925 14:54:01.235634 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.235635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.235646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.235699 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.236075 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.236829 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.237927 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.238226 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.238299 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.239411 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.239412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.239413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.239420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.241024 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.241435 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.241436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.241437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.241496 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.241705 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.241705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.241706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.241845 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.241916 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.242086 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.242163 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:01.242262 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.235375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:52096)
0925 14:54:01.235495 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.235508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.235578 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.235717 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.235718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.235920 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.235986 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.235996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.236926 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.236927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.237829 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.237832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.238312 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.238313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.238385 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.238902 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.238908 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.239319 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.241113 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.241113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.241114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.241328 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.241568 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.241568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.241569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.241636 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.241923 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.241924 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.241972 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.242046 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.242153 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260925 14:54:01.242878 27394 tablet_copy_service.cc:140] P 684c9a750c1148949f3cd4cc17d297cb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bb753be525354422a7b3d2a3d659bcbc ({username='oryx', principal='oryx/127.20.199.60@KRBTEST.COM'} at 127.0.0.1:52096)
I20260925 14:54:01.242952 27394 tablet_copy_service.cc:161] P 684c9a750c1148949f3cd4cc17d297cb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bb753be525354422a7b3d2a3d659bcbc at {username='oryx', principal='oryx/127.20.199.60@KRBTEST.COM'} at 127.0.0.1:52096: session id = bb753be525354422a7b3d2a3d659bcbc-00000000000000000000000000000000
I20260925 14:54:01.243507 27394 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:54:01.244016 28025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:54:01.245484 28025 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: tablet copy: Starting download of 0 data blocks...
I20260925 14:54:01.245643 28025 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: tablet copy: Starting download of 1 WAL segments...
I20260925 14:54:01.246392 28025 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:54:01.248593 27394 tablet_copy_service.cc:342] P 684c9a750c1148949f3cd4cc17d297cb: Request end of tablet copy session bb753be525354422a7b3d2a3d659bcbc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.199.60@KRBTEST.COM'} at 127.0.0.1:52096
I20260925 14:54:01.248651 27394 tablet_copy_service.cc:433] P 684c9a750c1148949f3cd4cc17d297cb: ending tablet copy session bb753be525354422a7b3d2a3d659bcbc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bb753be525354422a7b3d2a3d659bcbc
I20260925 14:54:01.251116 28025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:01.252242 28131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.252259 28130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.255056 28025 server_base.cc:1034] running on GCE node
W20260925 14:54:01.254984 28133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.255204 28025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:01.255352 28025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:01.256501 28025 hybrid_clock.cc:648] HybridClock initialized: now 1790348041256489 us; error 29 us; skew 500 ppm
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:01.258172 28025 init.cc:377] Logged in from keytab as oryx/127.20.199.60@KRBTEST.COM (short username oryx)
I20260925 14:54:01.258435 28025 webserver.cc:460] Webserver started at http://127.20.199.60:39893/ using document root <none> and password file <none>
I20260925 14:54:01.258556 28025 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:01.258590 28025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:01.259306 28025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:01.259948 28140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.260123 28025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:01.260174 28025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
uuid: "bb753be525354422a7b3d2a3d659bcbc"
format_stamp: "Formatted at 2026-09-25 14:54:01 on dist-test-slave-8qt6"
server_key: "eba26cb372e1e1517bc44babdcb7becb"
server_key_iv: "f8f77e9ecbf1527f1740ea32494c5aaa"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.260232 28025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:01.287073 28025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:01.289064 28025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:01.292253 28025 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.60:41853
I20260925 14:54:01.292599 28025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
I20260925 14:54:01.294891 28192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.60:41853 every 8 connection(s)
I20260925 14:54:01.295789 28193 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:54:01.296401 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Bootstrap starting.
I20260925 14:54:01.299636 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:54:01.299978 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Bootstrap complete.
I20260925 14:54:01.300246 28193 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } }
I20260925 14:54:01.300427 28193 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb753be525354422a7b3d2a3d659bcbc, State: Initialized, Role: LEARNER
I20260925 14:54:01.300509 28193 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } }
I20260925 14:54:01.300800 28193 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:01.300796 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.300995 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: This master's current role is: LEARNER
I20260925 14:54:01.302493 28207 catalog_manager.cc:1259] Loaded cluster ID: a9e4316ad58a45548f6d7685232580fe
I20260925 14:54:01.302541 28207 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: loading cluster ID for follower catalog manager: success
I20260925 14:54:01.303973 28207 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: acquiring CA information for follower catalog manager: success
I20260925 14:54:01.304271 28207 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:54:01.314213 28118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:01.314426 28118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:01.314451 28118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:01.315644 28118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:01.315685 28118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:01.315713 28118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:01.317191 28118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:38101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=38423
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:01.317595 28118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:01.317889 28118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:01.323220 28212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.323266 28213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.323496 28118 server_base.cc:1034] running on GCE node
W20260925 14:54:01.323233 28215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.324530 28118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:01.324858 28118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:01.326050 28118 hybrid_clock.cc:648] HybridClock initialized: now 1790348041326024 us; error 56 us; skew 500 ppm
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:01.329571 28118 init.cc:377] Logged in from keytab as oryx/127.20.199.1@KRBTEST.COM (short username oryx)
I20260925 14:54:01.330104 28118 webserver.cc:460] Webserver started at http://127.20.199.1:38423/ using document root <none> and password file <none>
I20260925 14:54:01.330358 28118 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:01.330422 28118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:01.331965 28118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:01.332644 28222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.332917 28118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:01.333039 28118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "ff97d18e016b4ca69b9e5f05af95ccf2"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "206161e9ebd43eb6f6e78ea4e4e89a4c"
server_key_iv: "6e33e18ddf48abcc90de78317a34a719"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.333454 28118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:01.350448 28118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:01.352618 28118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:01.352787 28118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:01.353045 28118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:01.353374 28118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:01.353456 28118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.353492 28118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:01.353520 28118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.359401 28118 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:38101
I20260925 14:54:01.359480 28335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:38101 every 8 connection(s)
I20260925 14:54:01.359922 28118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:01.363392 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 28118
I20260925 14:54:01.363545 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27570
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.60@KRBTEST.COM
I20260925 14:54:01.370012 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:44423
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:40765
--webserver_port=37165
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:01.373631 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.360720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38377 (local address 127.20.199.61:42661)
0925 14:54:01.360866 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.360868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.362440 (+ 1572us) server_negotiation.cc:407] Connection header received
0925 14:54:01.362519 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.362520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.362531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.362579 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.364032 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.364659 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.365948 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.366236 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.366279 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.369894 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.369896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.369897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.369906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.371750 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.372299 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.372301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.372301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.372373 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.372673 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.372674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.372674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.372834 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.372924 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.373467 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.373540 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 14:54:01.374078 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.361011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:59827)
0925 14:54:01.361491 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.361500 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.362304 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.362656 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.362659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.363174 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.363517 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.363524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.366216 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.366216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.367772 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.367775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.368355 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.368356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.368428 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.368937 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.368942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.370464 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.372778 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.372779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.372780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.373043 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.373253 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.373254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.373254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.373313 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.373652 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.373652 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.373690 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.373772 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.373877 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":39,"thread_start_us":56,"threads_started":1}
I20260925 14:54:01.374078 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.360701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.20.199.1:35229)
0925 14:54:01.361181 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.361197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.362294 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.363122 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.363123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.363295 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.363517 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.363524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.365834 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.365834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.366655 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.366657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.367203 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.367204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.367265 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.367645 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.367650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.369767 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.372793 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.372794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.372794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.373093 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.373335 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.373336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.373337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.373444 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.373775 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.373776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.373807 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.373870 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.373947 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260925 14:54:01.374183 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.360762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:35229 (local address 127.20.199.60:41853)
0925 14:54:01.360942 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.360945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.361289 (+ 344us) server_negotiation.cc:407] Connection header received
0925 14:54:01.363011 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.363013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.363029 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.363105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.363630 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.364652 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.366775 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.367080 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.367149 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.369898 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.369899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.369900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.369908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.372714 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.373218 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.373220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.373220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.373283 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.373541 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.373543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.373543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.373684 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.373742 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.373947 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.374046 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":52,"threads_started":1}
I20260925 14:54:01.374553 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.361129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59827 (local address 127.20.199.62:40765)
0925 14:54:01.361180 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.361181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.361564 (+ 383us) server_negotiation.cc:407] Connection header received
0925 14:54:01.362433 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.362435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.362449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.362482 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.365273 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.366089 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.367868 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.368224 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.368269 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.370587 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.370589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.370589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.370597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.372700 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.373124 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.373125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.373126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.373193 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.373391 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.373392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.373392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.373564 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.373626 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.374364 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.374450 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:54:01.374715 28336 heartbeater.cc:347] Connected to a master server at 127.20.199.60:41853
I20260925 14:54:01.374812 28336 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.374867 28337 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40765
I20260925 14:54:01.374905 28337 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.375031 28337 heartbeater.cc:511] Master 127.20.199.62:40765 requested a full tablet report, sending...
I20260925 14:54:01.375031 28336 heartbeater.cc:511] Master 127.20.199.60:41853 requested a full tablet report, sending...
I20260925 14:54:01.375495 27374 ts_manager.cc:194] Re-registered known tserver with Master: ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101)
I20260925 14:54:01.375564 28148 ts_manager.cc:194] Registered new tserver with Master: ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101)
I20260925 14:54:01.375950 27374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:59827
I20260925 14:54:01.377574 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.360701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.20.199.1:38377)
0925 14:54:01.361035 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.362394 (+ 1359us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.362446 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.362656 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.362659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.363159 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.363518 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.363526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.364772 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.364773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.365842 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.365843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.366369 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.366370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.366505 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.367123 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.367134 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.369767 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.371853 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.371868 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.371870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.372202 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.372450 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.372451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.372451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.372594 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.372961 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.372969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.373148 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.373255 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.373477 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":10,"thread_start_us":95,"threads_started":1}
I20260925 14:54:01.377938 28338 heartbeater.cc:347] Connected to a master server at 127.20.199.61:42661
I20260925 14:54:01.378000 28338 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.378072 28338 heartbeater.cc:511] Master 127.20.199.61:42661 requested a full tablet report, sending...
I20260925 14:54:01.378438 27971 ts_manager.cc:194] Registered new tserver with Master: ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101)
W20260925 14:54:01.457293 28344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:01.457587 28344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:01.457629 28344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:01.458870 28344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:01.458930 28344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:01.458981 28344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:01.460606 28344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:44423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=37165
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:01.460872 28344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:01.461154 28344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:01.466368 28352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.466367 28350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.466387 28349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.466509 28344 server_base.cc:1034] running on GCE node
I20260925 14:54:01.466741 28344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:01.466938 28344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:01.468103 28344 hybrid_clock.cc:648] HybridClock initialized: now 1790348041468095 us; error 46 us; skew 500 ppm
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:01.470723 28344 init.cc:377] Logged in from keytab as oryx/127.20.199.2@KRBTEST.COM (short username oryx)
I20260925 14:54:01.471161 28344 webserver.cc:460] Webserver started at http://127.20.199.2:37165/ using document root <none> and password file <none>
I20260925 14:54:01.471398 28344 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:01.471481 28344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:01.472781 28344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:01.473387 28359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.473675 28344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:01.473767 28344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "634f787031284d6f8d70941f91850b6d"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "60e6034c28f48401ac4a8a0221e9c878"
server_key_iv: "139a0f447f2bf1d6d600fbdd182f990e"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.474095 28344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:01.490020 28344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:01.492014 28344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:01.492123 28344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:01.492301 28344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:01.492615 28344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:01.492650 28344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.492694 28344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:01.492719 28344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.498466 28344 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44423
I20260925 14:54:01.498553 28472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44423 every 8 connection(s)
I20260925 14:54:01.498876 28344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.60@KRBTEST.COM
I20260925 14:54:01.508100 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 28344
I20260925 14:54:01.508266 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27704
I20260925 14:54:01.510178 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.500254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:54277 (local address 127.20.199.61:42661)
0925 14:54:01.500300 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.500302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.500447 (+ 145us) server_negotiation.cc:407] Connection header received
0925 14:54:01.500505 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.500506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.500518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.500573 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.501475 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.502225 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503352 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.503655 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503695 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.506225 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.506227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.506227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.506237 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.508062 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509004 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.509006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.509006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.509069 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509364 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.509365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.509365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.509545 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.509592 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.509997 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.510083 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:01.510392 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.500175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.20.199.2:54277)
0925 14:54:01.500356 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.500363 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.500444 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.501108 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.501109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.501253 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.501300 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.501306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.502315 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.502316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.503251 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.503253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.503774 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.503841 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.504334 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.504338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.506120 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.508158 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.508166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.508168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.508914 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.509145 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.509146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.509286 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.509640 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.509647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.509824 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.509959 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.510134 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260925 14:54:01.510205 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.499861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:36051 (local address 127.20.199.62:40765)
0925 14:54:01.499913 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.499916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.499927 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:01.500517 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.500518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.500530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.500562 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.501392 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.501960 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503061 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.503256 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503341 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.505756 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.505762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.505762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.505769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.508273 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509340 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.509341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.509341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.509437 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509648 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.509649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.509649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.509769 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.509812 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.510028 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.510116 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:54:01.510998 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.499601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:36051)
0925 14:54:01.499850 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.499861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.500382 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.500659 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.500660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.501094 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.501298 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.501302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.502080 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.502081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.502971 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.502974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.503349 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.503488 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.504051 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.504061 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.505640 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.509047 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.509048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.509252 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.509513 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.509513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.509514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.509574 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.509858 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.509858 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.509898 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.510820 (+ 922us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.510928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
I20260925 14:54:01.511611 28474 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40765
I20260925 14:54:01.511721 28474 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.511902 28474 heartbeater.cc:511] Master 127.20.199.62:40765 requested a full tablet report, sending...
I20260925 14:54:01.512316 27374 ts_manager.cc:194] Re-registered known tserver with Master: 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423)
I20260925 14:54:01.512704 27374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:36051
I20260925 14:54:01.513136 28475 heartbeater.cc:347] Connected to a master server at 127.20.199.61:42661
I20260925 14:54:01.513188 28475 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.513434 28475 heartbeater.cc:511] Master 127.20.199.61:42661 requested a full tablet report, sending...
I20260925 14:54:01.513751 27971 ts_manager.cc:194] Registered new tserver with Master: 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423)
I20260925 14:54:01.514170 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:44885
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:40765
--webserver_port=46203
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:01.514737 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.499600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.20.199.2:35793)
0925 14:54:01.499754 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.499796 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.500400 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.500652 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.500656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.501111 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.501300 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.501305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.502315 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.502316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.503267 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.503269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.503774 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.503841 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.504414 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.504418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.506730 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.508249 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.508250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.508250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.513894 (+ 5644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.514131 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.514132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.514132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.514187 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.514480 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.514481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.514515 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.514571 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.514661 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":53,"thread_start_us":108,"threads_started":1}
I20260925 14:54:01.514845 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.499693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:35793 (local address 127.20.199.60:41853)
0925 14:54:01.499918 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.499920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.499931 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:01.500499 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.500501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.500516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.500548 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.501443 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.502216 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503352 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.503656 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.503695 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.506827 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.506829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.506829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.506838 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.508149 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.513988 (+ 5839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.513990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.513990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.514069 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.514253 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.514254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.514255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.514417 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.514486 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.514596 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.514703 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260925 14:54:01.515389 28473 heartbeater.cc:347] Connected to a master server at 127.20.199.60:41853
I20260925 14:54:01.515470 28473 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.515551 28473 heartbeater.cc:511] Master 127.20.199.60:41853 requested a full tablet report, sending...
I20260925 14:54:01.515851 28148 ts_manager.cc:194] Registered new tserver with Master: 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423)
W20260925 14:54:01.595803 28480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:01.596021 28480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:01.596045 28480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:01.597208 28480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:01.597252 28480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:01.597278 28480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:01.598802 28480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:44885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=46203
--tserver_master_addrs=127.20.199.62:40765,127.20.199.61:42661,127.20.199.60:41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:01.599085 28480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:01.599336 28480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:54:01.604884 28480 server_base.cc:1034] running on GCE node
W20260925 14:54:01.604840 28486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.604858 28485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:01.604836 28488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:01.605250 28480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:01.605597 28480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:01.606755 28480 hybrid_clock.cc:648] HybridClock initialized: now 1790348041606744 us; error 32 us; skew 500 ppm
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:01.609589 28480 init.cc:377] Logged in from keytab as oryx/127.20.199.3@KRBTEST.COM (short username oryx)
I20260925 14:54:01.610116 28480 webserver.cc:460] Webserver started at http://127.20.199.3:46203/ using document root <none> and password file <none>
I20260925 14:54:01.610345 28480 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:01.610427 28480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:01.611783 28480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:01.612462 28495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.612663 28480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:01.612769 28480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "edec8472aec940b2995695543883ba11"
format_stamp: "Formatted at 2026-09-25 14:54:00 on dist-test-slave-8qt6"
server_key: "5120acf814054623d0659a43584e8522"
server_key_iv: "eeca7f6455c86d5bba020cf21ec23288"
server_key_version: "encryptionkey@0"
I20260925 14:54:01.613147 28480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:01.631960 28480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:01.634415 28480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:01.634589 28480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:01.634838 28480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:01.635159 28480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:01.635212 28480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.635255 28480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:01.635283 28480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:01.642319 28480 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:44885
I20260925 14:54:01.642410 28608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:44885 every 8 connection(s)
I20260925 14:54:01.642827 28480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:01.649502 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 28480
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.61@KRBTEST.COM
Sep 25 14:54:01 dist-test-slave-8qt6 krb5kdc[27329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348041, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.60@KRBTEST.COM
I20260925 14:54:01.656831 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.643680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.20.199.3:43237)
0925 14:54:01.643876 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.643916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.645090 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.645384 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.645389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.645883 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.646054 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.646060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.647150 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.647150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.648165 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.648168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.648685 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.648772 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.649354 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.649361 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.652074 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.654984 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.654985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.654985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.655188 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.655380 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.655380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.655515 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.656061 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.656062 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.656210 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.656344 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.656552 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":61,"thread_start_us":98,"threads_started":1}
I20260925 14:54:01.657119 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.644014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:55343)
0925 14:54:01.645339 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.645351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.645442 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.645588 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.645588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.645897 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.646054 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.646060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.647145 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.647146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.648174 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.648176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.648662 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.648735 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.649103 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.649107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.652615 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.655178 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.655180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.655416 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.655630 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.655630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.655699 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.656013 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.656021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.656210 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.656821 (+ 611us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.656952 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260925 14:54:01.657593 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.643680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.20.199.3:57243)
0925 14:54:01.644144 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.644156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.645110 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.645383 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.645388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.645833 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:01.646054 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.646058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.646753 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.646754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.647722 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.647725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.648165 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.648319 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.648975 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:01.648984 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:01.651663 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:01.653816 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.653824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.653826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.654140 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.655721 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:01.655722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:01.655776 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:01.657213 (+ 1437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:01.657214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:01.657263 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.657333 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.657461 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260925 14:54:01.657981 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.643686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:43237 (local address 127.20.199.60:41853)
0925 14:54:01.643737 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.643740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.644021 (+ 281us) server_negotiation.cc:407] Connection header received
0925 14:54:01.645217 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.645219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.645233 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.645299 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.646193 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.647031 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648287 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.648519 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648624 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.652176 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.652177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.652178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.652185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.654890 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655264 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.655265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.655266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.655318 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655840 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.655841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.655841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.655974 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.656033 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.657774 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.657865 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:01.658494 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.643747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:57243 (local address 127.20.199.61:42661)
0925 14:54:01.643793 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.643795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.644214 (+ 419us) server_negotiation.cc:407] Connection header received
0925 14:54:01.645224 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.645226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.645240 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.645301 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.646155 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.646646 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.647839 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.648056 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648091 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.651825 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.651827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.651827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.651834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.653721 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655590 (+ 1869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.655591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.655591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.655650 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.656963 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.656965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.656965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.657117 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.657181 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.658295 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.658395 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:01.658520 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.644229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:55343 (local address 127.20.199.62:40765)
0925 14:54:01.644287 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.644289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.645423 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 14:54:01.645515 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.645517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.645530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.645580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.646185 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.647032 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648261 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.648563 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.648660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.652732 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.652733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.652733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.653209 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.655081 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655497 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.655498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.655498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.655558 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.655771 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.655772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.655772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.655922 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.655985 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.658316 (+ 2331us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.658426 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:54:01.658630 28609 heartbeater.cc:347] Connected to a master server at 127.20.199.60:41853
I20260925 14:54:01.658727 28609 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.658823 28611 heartbeater.cc:347] Connected to a master server at 127.20.199.61:42661
I20260925 14:54:01.658852 28611 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.658931 28609 heartbeater.cc:511] Master 127.20.199.60:41853 requested a full tablet report, sending...
I20260925 14:54:01.658931 28611 heartbeater.cc:511] Master 127.20.199.61:42661 requested a full tablet report, sending...
I20260925 14:54:01.659128 28610 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40765
I20260925 14:54:01.659157 28610 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:01.659222 28610 heartbeater.cc:511] Master 127.20.199.62:40765 requested a full tablet report, sending...
I20260925 14:54:01.659391 27971 ts_manager.cc:194] Registered new tserver with Master: edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:01.659438 27374 ts_manager.cc:194] Re-registered known tserver with Master: edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:01.659500 28148 ts_manager.cc:194] Registered new tserver with Master: edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:01.659906 27374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:55343
I20260925 14:54:01.662533 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.652045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.20.199.62:40765)
0925 14:54:01.652473 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.652475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.652966 (+ 491us) server_negotiation.cc:407] Connection header received
0925 14:54:01.652990 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.652991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.653006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.653071 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.653817 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.654589 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.655692 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.656033 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.656094 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.657816 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.657818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.657818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.657825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.661168 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.661656 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.661657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.661658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.661759 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.662001 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.662002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.662002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.662122 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.662231 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.662341 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.662416 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":396,"thread_start_us":80,"threads_started":1}
I20260925 14:54:01.663975 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.650940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.20.199.60:41853)
0925 14:54:01.651080 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.651082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.651091 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:01.651113 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.651114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.651126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.651228 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.653520 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.654463 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.655722 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.656054 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.656097 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.660591 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.660593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.660593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.660600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.662585 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.663080 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.663081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.663081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.663144 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.663365 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.663366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.663366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.663509 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.663585 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.663746 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.663848 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260925 14:54:01.665930 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.652541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.20.199.61:42661)
0925 14:54:01.652791 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.652793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.656465 (+ 3672us) server_negotiation.cc:407] Connection header received
0925 14:54:01.656524 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.656526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.656539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.656608 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:01.658431 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.659481 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.660542 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.660902 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.660977 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.662489 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:01.662490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:01.662491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:01.662499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:01.664643 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.664985 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.664985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.664986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.665053 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:01.665278 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:01.665278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:01.665279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:01.665451 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:01.665559 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.665683 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.665753 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260925 14:54:01.664479 27374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:01.666409 27374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:01.674608 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.670912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.20.199.2:44423)
0925 14:54:01.671080 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.671086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.671098 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:54:01.671141 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.671144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.671161 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.671425 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:01.671489 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.672672 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.673789 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.674260 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.674350 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.674432 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.674486 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.674522 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260925 14:54:01.674611 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.670468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44423 (local address 127.0.0.1:35572)
0925 14:54:01.670531 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.670545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.670613 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.671285 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.671286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.671288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:01.671338 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.671340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.672763 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.672764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.673709 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.673711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.674349 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.674350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.674406 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.674409 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.674468 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.674505 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:54:01.674997 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.670573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:44885 (local address 127.0.0.1:58348)
0925 14:54:01.670729 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.670738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.670779 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.671264 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.671265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.671268 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:01.671344 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.671346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.672327 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.672328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.673325 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.673327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.674389 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.674390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.674451 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.674456 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.674488 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.674529 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260925 14:54:01.675136 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.670898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.20.199.3:44885)
0925 14:54:01.671059 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.671063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.671073 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:01.671118 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.671119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.671133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.671454 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:01.671511 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.672231 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.673449 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.674259 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.674659 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.674896 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.674925 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.674968 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260925 14:54:01.675354 28407 tablet_service.cc:1467] Processing CreateTablet for tablet 60114d1ff0ec414ca818b975007717f3 (DEFAULT_TABLE table=test-table [id=16aecb315fb14220bb4bd1684ec2573e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:01.675711 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.670617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.20.199.1:38101)
0925 14:54:01.670891 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.670896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.671276 (+ 380us) server_negotiation.cc:407] Connection header received
0925 14:54:01.671392 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.671395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.671413 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.671464 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:01.671678 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.672838 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.674386 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.675204 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.675261 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.675336 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.675579 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.675629 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20260925 14:54:01.675727 28407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60114d1ff0ec414ca818b975007717f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:01.675943 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.670468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:38101 (local address 127.0.0.1:57146)
0925 14:54:01.671212 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.671222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.671301 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.671550 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.671551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.671553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:01.671593 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.671594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.672963 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.672964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.674268 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.674271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.675406 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.675407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.675477 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.675480 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.675828 (+ 348us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.675878 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":594,"thread_start_us":79,"threads_started":1}
I20260925 14:54:01.676751 28270 tablet_service.cc:1467] Processing CreateTablet for tablet 60114d1ff0ec414ca818b975007717f3 (DEFAULT_TABLE table=test-table [id=16aecb315fb14220bb4bd1684ec2573e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:01.677145 28270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60114d1ff0ec414ca818b975007717f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:01.678050 28543 tablet_service.cc:1467] Processing CreateTablet for tablet 60114d1ff0ec414ca818b975007717f3 (DEFAULT_TABLE table=test-table [id=16aecb315fb14220bb4bd1684ec2573e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:01.678400 28543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60114d1ff0ec414ca818b975007717f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:01.679061 28635 tablet_bootstrap.cc:492] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Bootstrap starting.
I20260925 14:54:01.679822 28635 tablet_bootstrap.cc:654] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:01.680177 28635 log.cc:824] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:01.680951 28637 tablet_bootstrap.cc:492] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Bootstrap starting.
I20260925 14:54:01.680953 28635 tablet_bootstrap.cc:492] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: No bootstrap required, opened a new log
I20260925 14:54:01.681092 28635 ts_tablet_manager.cc:1397] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:01.681702 28637 tablet_bootstrap.cc:654] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:01.682037 28637 log.cc:824] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:01.682209 28638 tablet_bootstrap.cc:492] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Bootstrap starting.
I20260925 14:54:01.682823 28638 tablet_bootstrap.cc:654] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:01.683128 28638 log.cc:824] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:01.683867 28638 tablet_bootstrap.cc:492] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: No bootstrap required, opened a new log
I20260925 14:54:01.683964 28638 ts_tablet_manager.cc:1397] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:01.683928 28635 raft_consensus.cc:348] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:01.684109 28635 raft_consensus.cc:374] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:01.684170 28635 raft_consensus.cc:729] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634f787031284d6f8d70941f91850b6d, State: Initialized, Role: FOLLOWER
I20260925 14:54:01.684285 28635 consensus_queue.cc:260] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:01.684587 28474 heartbeater.cc:503] Master 127.20.199.62:40765 was elected leader, sending a full tablet report...
I20260925 14:54:01.684544 28635 ts_tablet_manager.cc:1428] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:01.685808 28637 tablet_bootstrap.cc:492] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: No bootstrap required, opened a new log
I20260925 14:54:01.685881 28637 ts_tablet_manager.cc:1397] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 14:54:01.687755 28638 raft_consensus.cc:348] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:01.687927 28638 raft_consensus.cc:374] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:01.687966 28638 raft_consensus.cc:729] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edec8472aec940b2995695543883ba11, State: Initialized, Role: FOLLOWER
I20260925 14:54:01.688087 28638 consensus_queue.cc:260] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:01.688378 28638 ts_tablet_manager.cc:1428] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 14:54:01.688417 28610 heartbeater.cc:503] Master 127.20.199.62:40765 was elected leader, sending a full tablet report...
I20260925 14:54:01.688367 28637 raft_consensus.cc:348] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:01.688524 28637 raft_consensus.cc:374] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:01.688562 28637 raft_consensus.cc:729] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff97d18e016b4ca69b9e5f05af95ccf2, State: Initialized, Role: FOLLOWER
I20260925 14:54:01.688676 28637 consensus_queue.cc:260] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:01.688962 28637 ts_tablet_manager.cc:1428] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:54:01.689078 28337 heartbeater.cc:503] Master 127.20.199.62:40765 was elected leader, sending a full tablet report...
I20260925 14:54:01.705474 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.702665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:33500)
0925 14:54:01.702747 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:01.702766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:01.702827 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:01.703195 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:01.703196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:01.703198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:01.703245 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.703247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.703877 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.703878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.704614 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:01.704616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.705224 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.705224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:01.705268 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.705272 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:01.705313 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 14:54:01.705348 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 14:54:01.705472 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:01.702783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.20.199.60:41853)
0925 14:54:01.702850 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:01.702852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:01.702873 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:54:01.703090 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:01.703091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:01.703112 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:01.703181 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:01.703313 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.703795 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.704706 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:01.705148 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:01.705213 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:01.705269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:01.705316 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:54:01.705345 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:01.705845 28167 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 14:54:01.706755 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 684c9a750c1148949f3cd4cc17d297cb. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:01.706914 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: This master's current role is: LEARNER
W20260925 14:54:01.749651 28476 tablet.cc:2369] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:01.860972 28339 tablet.cc:2369] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:01.894207 28612 tablet.cc:2369] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:02.087785 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: NON_VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: true } } }
I20260925 14:54:02.087947 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:02.088287 27932 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: attempting to promote NON_VOTER bb753be525354422a7b3d2a3d659bcbc to VOTER
I20260925 14:54:02.088431 27932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:02.088907 28167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 LEARNER]: Refusing update from remote peer 684c9a750c1148949f3cd4cc17d297cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 14:54:02.088908 27981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Refusing update from remote peer 684c9a750c1148949f3cd4cc17d297cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 14:54:02.089172 27933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 14:54:02.089324 28123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 14:54:02.090123 28123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, bb753be525354422a7b3d2a3d659bcbc (127.20.199.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.090271 27981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, bb753be525354422a7b3d2a3d659bcbc (127.20.199.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.090299 28167 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, bb753be525354422a7b3d2a3d659bcbc (127.20.199.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.090996 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.091068 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:02.091125 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.091172 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:02.091212 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.091260 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:02.091368 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.091461 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:02.091611 28649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "684c9a750c1148949f3cd4cc17d297cb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:02.091703 28649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:02.115847 28643 raft_consensus.cc:482] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:02.116005 28643 raft_consensus.cc:504] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.116393 28643 leader_election.cc:288] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423), edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:02.117264 28641 raft_consensus.cc:482] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:02.117390 28641 raft_consensus.cc:504] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.117756 28641 leader_election.cc:288] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101), edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:02.120446 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.116983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59559 (local address 127.20.199.3:44885)
0925 14:54:02.117040 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.117043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.117066 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:54:02.117125 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.117126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.117136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.117188 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:02.117354 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.118087 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.119559 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.120074 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.120121 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.120130 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.120326 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.120364 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:54:02.120688 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.116699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44423 (local address 127.20.199.1:60633)
0925 14:54:02.116848 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:02.116860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:02.116926 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:02.117088 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:02.117089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:02.117091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:02.117168 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.117170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.118636 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.118637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.119731 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.119733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.120431 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.120431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.120498 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.120502 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:02.120544 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:54:02.120584 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260925 14:54:02.120901 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.116789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:60633 (local address 127.20.199.2:44423)
0925 14:54:02.116853 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.116856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.116968 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:54:02.116999 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.117000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.117011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.117080 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:02.117873 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.118551 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.119816 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.120326 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.120391 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.120400 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.120795 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.120841 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:02.121105 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.116915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:44885 (local address 127.20.199.1:59559)
0925 14:54:02.117017 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:02.117027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:02.117059 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:02.117204 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:02.117205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:02.117207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:02.117263 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.117265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.118170 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.118172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.119467 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.119470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.120187 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.120188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.120249 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.120253 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:02.120715 (+ 462us) client_negotiation.cc:240] Negotiation successful
0925 14:54:02.120909 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260925 14:54:02.121129 28427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634f787031284d6f8d70941f91850b6d" is_pre_election: true
I20260925 14:54:02.121238 28427 raft_consensus.cc:2454] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 0.
I20260925 14:54:02.121572 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.117912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:38101 (local address 127.20.199.2:38423)
0925 14:54:02.118358 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:02.118365 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:02.118428 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:02.118591 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:02.118592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:02.118594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:02.118640 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.118641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.119498 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.119499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.120590 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.120592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.121280 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.121281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.121370 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.121375 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:02.121466 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:54:02.121507 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":41,"threads_started":1}
I20260925 14:54:02.121570 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.118118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:38423 (local address 127.20.199.1:38101)
0925 14:54:02.118157 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.118159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.118409 (+ 250us) server_negotiation.cc:407] Connection header received
0925 14:54:02.118489 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.118490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.118497 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.118531 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:02.118721 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.119405 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.120674 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.121171 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.121217 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.121224 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.121487 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.121520 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":21}
I20260925 14:54:02.121628 28225 leader_election.cc:302] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634f787031284d6f8d70941f91850b6d, ff97d18e016b4ca69b9e5f05af95ccf2; no voters:
I20260925 14:54:02.121636 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edec8472aec940b2995695543883ba11" is_pre_election: true
I20260925 14:54:02.121766 28563 raft_consensus.cc:2454] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 0.
I20260925 14:54:02.121785 28643 raft_consensus.cc:2781] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:02.121863 28643 raft_consensus.cc:482] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:02.121888 28643 raft_consensus.cc:3037] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:02.121917 28290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "634f787031284d6f8d70941f91850b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" is_pre_election: true
I20260925 14:54:02.122282 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.117886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:44885 (local address 127.20.199.2:46373)
0925 14:54:02.118209 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:02.118219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:02.118270 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:02.118445 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:02.118446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:02.118449 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:02.118492 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.118494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.119389 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.119390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.120586 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:02.120589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.122070 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.122071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:02.122136 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.122140 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:02.122171 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:54:02.122211 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":56,"threads_started":1}
I20260925 14:54:02.122598 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.118029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:46373 (local address 127.20.199.3:44885)
0925 14:54:02.118121 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.118123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.118279 (+ 156us) server_negotiation.cc:407] Connection header received
0925 14:54:02.118314 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.118315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.118338 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.118551 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:02.118582 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.119295 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.120875 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.121498 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.122485 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.122495 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.122519 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.122547 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":53,"threads_started":1}
I20260925 14:54:02.122747 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "634f787031284d6f8d70941f91850b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edec8472aec940b2995695543883ba11" is_pre_election: true
I20260925 14:54:02.122758 28643 raft_consensus.cc:504] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.122829 28563 raft_consensus.cc:2454] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634f787031284d6f8d70941f91850b6d in term 0.
I20260925 14:54:02.122907 28290 raft_consensus.cc:2379] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 634f787031284d6f8d70941f91850b6d in current term 1: Already voted for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in this term.
I20260925 14:54:02.122917 28643 leader_election.cc:288] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 election: Requested vote from peers 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423), edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:02.123018 28362 leader_election.cc:302] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634f787031284d6f8d70941f91850b6d, edec8472aec940b2995695543883ba11; no voters:
I20260925 14:54:02.123086 28427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634f787031284d6f8d70941f91850b6d"
I20260925 14:54:02.123131 28641 raft_consensus.cc:2781] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:02.123185 28427 raft_consensus.cc:3037] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:02.123159 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edec8472aec940b2995695543883ba11"
I20260925 14:54:02.123219 28563 raft_consensus.cc:3037] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:02.123847 28427 raft_consensus.cc:2454] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 1.
I20260925 14:54:02.123934 28641 raft_consensus.cc:482] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:02.123980 28641 raft_consensus.cc:3037] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:54:02.123972 28563 raft_consensus.cc:2454] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 1.
I20260925 14:54:02.124142 28225 leader_election.cc:302] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edec8472aec940b2995695543883ba11, ff97d18e016b4ca69b9e5f05af95ccf2; no voters:
I20260925 14:54:02.124248 28643 raft_consensus.cc:2781] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:02.124373 28643 raft_consensus.cc:686] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 LEADER]: Becoming Leader. State: Replica: ff97d18e016b4ca69b9e5f05af95ccf2, State: Running, Role: LEADER
I20260925 14:54:02.124466 28643 consensus_queue.cc:237] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.124727 28641 raft_consensus.cc:504] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.124864 28641 leader_election.cc:288] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [CANDIDATE]: Term 2 election: Requested vote from peers ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101), edec8472aec940b2995695543883ba11 (127.20.199.3:44885)
I20260925 14:54:02.125264 27373 catalog_manager.cc:5495] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1). New cstate: current_term: 1 leader_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:02.125514 28290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "634f787031284d6f8d70941f91850b6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2"
I20260925 14:54:02.125604 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60114d1ff0ec414ca818b975007717f3" candidate_uuid: "634f787031284d6f8d70941f91850b6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edec8472aec940b2995695543883ba11"
I20260925 14:54:02.125667 28563 raft_consensus.cc:3037] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:54:02.126199 28563 raft_consensus.cc:2454] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634f787031284d6f8d70941f91850b6d in term 2.
I20260925 14:54:02.126447 28362 leader_election.cc:302] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634f787031284d6f8d70941f91850b6d, edec8472aec940b2995695543883ba11; no voters: ff97d18e016b4ca69b9e5f05af95ccf2
I20260925 14:54:02.126685 28641 raft_consensus.cc:2781] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:54:02.126794 28641 raft_consensus.cc:686] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 2 LEADER]: Becoming Leader. State: Replica: 634f787031284d6f8d70941f91850b6d, State: Running, Role: LEADER
I20260925 14:54:02.126904 28641 consensus_queue.cc:237] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.127544 27374 catalog_manager.cc:5495] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d reported cstate change: term changed from 1 to 2, leader changed from ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1) to 634f787031284d6f8d70941f91850b6d (127.20.199.2). New cstate: current_term: 2 leader_uuid: "634f787031284d6f8d70941f91850b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:02.223734 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.221228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.20.199.2:44423)
0925 14:54:02.221307 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.221309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.221326 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:54:02.221436 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.221437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.221463 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.221541 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:02.221684 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.222160 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.222946 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.223148 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.223225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.223392 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:02.223499 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:02.223587 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.223616 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.223634 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:54:02.225353 28290 raft_consensus.cc:3032] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 14:54:02.225447 28290 raft_consensus.cc:729] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff97d18e016b4ca69b9e5f05af95ccf2, State: Running, Role: LEADER
I20260925 14:54:02.225469 28563 raft_consensus.cc:1261] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 2 FOLLOWER]: Refusing update from remote peer 634f787031284d6f8d70941f91850b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 14:54:02.225512 28290 consensus_queue.cc:260] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } }
I20260925 14:54:02.225625 28290 raft_consensus.cc:3037] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:54:02.225775 28657 consensus_queue.cc:1035] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:02.226553 28290 raft_consensus.cc:1261] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 2 FOLLOWER]: Refusing update from remote peer 634f787031284d6f8d70941f91850b6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 14:54:02.226827 28657 consensus_queue.cc:1035] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:02.226910 28657 consensus_queue.cc:1225] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [LEADER]: Peer ff97d18e016b4ca69b9e5f05af95ccf2 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260925 14:54:02.227280 28290 pending_rounds.cc:76] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Aborting all ops after (but not including) 0
I20260925 14:54:02.227337 28290 pending_rounds.cc:97] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260925 14:54:02.227385 28290 raft_consensus.cc:2866] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 14:54:02.228811 28665 mvcc.cc:205] Tried to move back new op lower bound from 7333265580952776704 to 7333265580552409088. Current Snapshot: MvccSnapshot[applied={T|T < 7333265580952776704}]
I20260925 14:54:02.229815 28661 mvcc.cc:205] Tried to move back new op lower bound from 7333265580952776704 to 7333265580552409088. Current Snapshot: MvccSnapshot[applied={T|T < 7333265580952776704}]
I20260925 14:54:02.231026 28663 mvcc.cc:205] Tried to move back new op lower bound from 7333265580952776704 to 7333265580552409088. Current Snapshot: MvccSnapshot[applied={T|T < 7333265580952776704 or (T in {7333265580952776704})}]
I20260925 14:54:02.232411 27374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:02.232508 27374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:02.233821 27374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 684c9a750c1148949f3cd4cc17d297cb: Sending DeleteTablet for 3 replicas of tablet 60114d1ff0ec414ca818b975007717f3
I20260925 14:54:02.234174 28270 tablet_service.cc:1514] Processing DeleteTablet for tablet 60114d1ff0ec414ca818b975007717f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:02 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:57146
I20260925 14:54:02.234257 28407 tablet_service.cc:1514] Processing DeleteTablet for tablet 60114d1ff0ec414ca818b975007717f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:02 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:35572
I20260925 14:54:02.234288 28543 tablet_service.cc:1514] Processing DeleteTablet for tablet 60114d1ff0ec414ca818b975007717f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:02 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:58348
I20260925 14:54:02.234660 28670 tablet_replica.cc:330] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: stopping tablet replica
I20260925 14:54:02.234771 28670 raft_consensus.cc:2229] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:02.234894 28670 raft_consensus.cc:2258] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:02.235085 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27342
I20260925 14:54:02.235309 28670 ts_tablet_manager.cc:1905] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:02.235509 28672 tablet_replica.cc:330] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: stopping tablet replica
I20260925 14:54:02.235661 28672 raft_consensus.cc:2229] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:02.235852 28672 raft_consensus.cc:2258] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:02.236263 28672 ts_tablet_manager.cc:1905] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:02.237053 28670 ts_tablet_manager.cc:1918] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260925 14:54:02.237135 28670 log.cc:1196] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/60114d1ff0ec414ca818b975007717f3
I20260925 14:54:02.237442 28670 ts_tablet_manager.cc:1939] T 60114d1ff0ec414ca818b975007717f3 P ff97d18e016b4ca69b9e5f05af95ccf2: Deleting consensus metadata
I20260925 14:54:02.238277 28671 tablet_replica.cc:330] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: stopping tablet replica
I20260925 14:54:02.238329 28672 ts_tablet_manager.cc:1918] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260925 14:54:02.238401 28672 log.cc:1196] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/60114d1ff0ec414ca818b975007717f3
I20260925 14:54:02.238533 28671 raft_consensus.cc:2229] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 2 LEADER]: Raft consensus shutting down.
I20260925 14:54:02.238659 28672 ts_tablet_manager.cc:1939] T 60114d1ff0ec414ca818b975007717f3 P edec8472aec940b2995695543883ba11: Deleting consensus metadata
I20260925 14:54:02.238741 28671 raft_consensus.cc:2258] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:02.239109 28671 ts_tablet_manager.cc:1905] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:02.240348 28671 ts_tablet_manager.cc:1918] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260925 14:54:02.240449 28671 log.cc:1196] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/60114d1ff0ec414ca818b975007717f3
I20260925 14:54:02.240688 28671 ts_tablet_manager.cc:1939] T 60114d1ff0ec414ca818b975007717f3 P 634f787031284d6f8d70941f91850b6d: Deleting consensus metadata
W20260925 14:54:02.242491 28497 connection.cc:535] server connection from 127.0.0.1:58348 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:54:02.242718 28360 connection.cc:535] server connection from 127.0.0.1:35572 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:54:02.243050 28226 connection.cc:535] server connection from 127.0.0.1:57146 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:54:02.249135 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.241638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.20.199.60:41853)
0925 14:54:02.241763 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.241765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.241773 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:02.241794 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.241794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.241804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.241919 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:02.242491 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.243231 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.244049 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.244303 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.244371 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.246043 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:02.246044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:02.246045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:02.246052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:02.247821 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:02.248223 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:02.248224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:02.248225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:02.248283 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:02.248477 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:02.248478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:02.248478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:02.248601 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:02.248689 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.248950 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.249031 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260925 14:54:02.253106 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.244761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.20.199.60:41853)
0925 14:54:02.244993 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.244995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.245165 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:54:02.245188 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.245188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.245202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.245258 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:02.245966 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.246735 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.247817 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.248110 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.248174 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.250172 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:02.250173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:02.250174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:02.250180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:02.251892 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:02.252254 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:02.252256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:02.252256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:02.252313 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:02.252548 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:02.252549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:02.252550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:02.252670 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:02.252765 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.252919 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.253004 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":45,"threads_started":1}
I20260925 14:54:02.254422 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:02.245249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.20.199.61:42661)
0925 14:54:02.245485 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:02.245486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:02.245676 (+ 190us) server_negotiation.cc:407] Connection header received
0925 14:54:02.245777 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:02.245778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:02.245791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:02.245831 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:02.246710 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.247745 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.249052 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:02.249444 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:02.249494 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:02.251495 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:02.251497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:02.251497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:02.251504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:02.253346 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:02.253747 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:02.253748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:02.253748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:02.253812 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:02.254000 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:02.254000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:02.254000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:02.254100 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:02.254169 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:02.254298 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:54:02.254357 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":56,"threads_started":1}
I20260925 14:54:03.230454 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.230122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:59135)
0925 14:54:03.230266 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.230374 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260925 14:54:03.230719 28337 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:40765 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.231132 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.230877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:38645)
0925 14:54:03.231019 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.231089 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
W20260925 14:54:03.231424 28474 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:40765 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.231570 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.231361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:51017)
0925 14:54:03.231460 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.231528 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
W20260925 14:54:03.231770 28610 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:40765 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.734400 28688 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 684c9a750c1148949f3cd4cc17d297cb)
I20260925 14:54:03.734400 28687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 684c9a750c1148949f3cd4cc17d297cb)
I20260925 14:54:03.734529 28688 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:03.734566 28687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:03.734894 28687 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765), bb753be525354422a7b3d2a3d659bcbc (127.20.199.60:41853)
I20260925 14:54:03.734874 28688 leader_election.cc:288] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765), 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61:42661)
I20260925 14:54:03.735316 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.735028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50504)
0925 14:54:03.735202 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.735268 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":57,"threads_started":1}
W20260925 14:54:03.735482 27958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.735349 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.735004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50502)
0925 14:54:03.735267 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.735310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":53,"threads_started":1}
W20260925 14:54:03.735929 28144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.736374 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.736259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50530)
0925 14:54:03.736311 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.736345 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:54:03.736509 27958 leader_election.cc:334] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.736603 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.736358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50546)
0925 14:54:03.736511 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.736543 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
W20260925 14:54:03.736722 28144 leader_election.cc:334] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.738775 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.735031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:41853 (local address 127.0.0.1:51552)
0925 14:54:03.735394 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.735406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:03.735505 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:03.735735 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:03.735736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:03.735739 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:03.735794 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:03.735796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.736912 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.736913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:03.737940 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:03.737943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.738477 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.738478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:03.738548 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:03.738556 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:54:03.738628 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:54:03.738687 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":48,"threads_started":1}
I20260925 14:54:03.738787 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.735237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.20.199.60:41853)
0925 14:54:03.735319 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:03.735322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:03.735473 (+ 151us) server_negotiation.cc:407] Connection header received
0925 14:54:03.735611 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:03.735612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:03.735622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:03.735698 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:03.735929 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.736792 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.738036 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.738395 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.738448 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:03.738458 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:03.738661 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 14:54:03.738705 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":36,"threads_started":1}
I20260925 14:54:03.738791 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.735252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.20.199.61:42661)
0925 14:54:03.735368 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:03.735369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:03.735421 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:54:03.735491 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:03.735494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:03.735508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:03.735586 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:03.735832 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.736618 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.738051 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.738477 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.738549 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:03.738557 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:03.738692 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 14:54:03.738726 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":44,"threads_started":1}
I20260925 14:54:03.738792 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.735162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42661 (local address 127.0.0.1:50356)
0925 14:54:03.735333 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.735344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:03.735421 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:03.735690 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:03.735691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:03.735694 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:03.735750 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:03.735753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.736715 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.736716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:03.737940 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:03.737943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:03.738556 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:03.738556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:03.738595 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:03.738616 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:54:03.738675 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:54:03.738707 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":50,"threads_started":1}
I20260925 14:54:03.739138 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb753be525354422a7b3d2a3d659bcbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" is_pre_election: true
I20260925 14:54:03.739140 28167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "bb753be525354422a7b3d2a3d659bcbc" is_pre_election: true
I20260925 14:54:03.739261 28167 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3d21a05a5941289d00c3aa6d0a28e9 in term 1.
I20260925 14:54:03.739243 27981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb753be525354422a7b3d2a3d659bcbc in term 1.
I20260925 14:54:03.739428 27955 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f3d21a05a5941289d00c3aa6d0a28e9, bb753be525354422a7b3d2a3d659bcbc; no voters: 684c9a750c1148949f3cd4cc17d297cb
I20260925 14:54:03.739429 28144 leader_election.cc:302] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f3d21a05a5941289d00c3aa6d0a28e9, bb753be525354422a7b3d2a3d659bcbc; no voters: 684c9a750c1148949f3cd4cc17d297cb
I20260925 14:54:03.739521 28687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:54:03.739542 28688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:54:03.739563 28687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 684c9a750c1148949f3cd4cc17d297cb)
I20260925 14:54:03.739573 28688 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 684c9a750c1148949f3cd4cc17d297cb)
I20260925 14:54:03.739598 28688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:54:03.739598 28687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:54:03.740276 28687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:03.740300 28688 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:03.740425 28688 leader_election.cc:288] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 2 election: Requested vote from peers 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765), 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61:42661)
I20260925 14:54:03.740419 28687 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [CANDIDATE]: Term 2 election: Requested vote from peers 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765), bb753be525354422a7b3d2a3d659bcbc (127.20.199.60:41853)
I20260925 14:54:03.740598 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.740488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50558)
0925 14:54:03.740528 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.740561 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 14:54:03.740697 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb753be525354422a7b3d2a3d659bcbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
I20260925 14:54:03.740697 28167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "bb753be525354422a7b3d2a3d659bcbc"
I20260925 14:54:03.740787 27981 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bb753be525354422a7b3d2a3d659bcbc in current term 2: Already voted for candidate 3f3d21a05a5941289d00c3aa6d0a28e9 in this term.
I20260925 14:54:03.740787 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.740560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50568)
0925 14:54:03.740619 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.740695 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:54:03.740833 28167 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3f3d21a05a5941289d00c3aa6d0a28e9 in current term 2: Already voted for candidate bb753be525354422a7b3d2a3d659bcbc in this term.
I20260925 14:54:03.740907 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.740815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50580)
0925 14:54:03.740853 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.740878 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 14:54:03.740972 27958 leader_election.cc:334] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.741279 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:03.741180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50584)
0925 14:54:03.741230 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.741257 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:54:03.741273 27955 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f3d21a05a5941289d00c3aa6d0a28e9; no voters: 684c9a750c1148949f3cd4cc17d297cb, bb753be525354422a7b3d2a3d659bcbc
W20260925 14:54:03.741374 28144 leader_election.cc:334] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:03.741444 28687 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 14:54:03.741453 28144 leader_election.cc:302] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb753be525354422a7b3d2a3d659bcbc; no voters: 3f3d21a05a5941289d00c3aa6d0a28e9, 684c9a750c1148949f3cd4cc17d297cb
I20260925 14:54:03.741554 28688 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 14:54:04.231379 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.231063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:55083)
0925 14:54:04.231263 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.231316 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":109,"threads_started":1}
I20260925 14:54:04.232095 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.231773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:54599)
0925 14:54:04.231980 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.232024 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
I20260925 14:54:04.232357 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.232048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:42597)
0925 14:54:04.232286 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.232320 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260925 14:54:05.232160 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.231923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:35975)
0925 14:54:05.232052 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.232096 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260925 14:54:05.232750 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.232511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:33925)
0925 14:54:05.232663 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.232701 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260925 14:54:05.232945 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.232712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:58743)
0925 14:54:05.232825 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.232900 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260925 14:54:05.426237 28704 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:05.426352 28704 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:05.426538 28704 leader_election.cc:288] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765), 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61:42661)
I20260925 14:54:05.426909 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.426687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50806)
0925 14:54:05.426788 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.426828 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260925 14:54:05.426930 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb753be525354422a7b3d2a3d659bcbc" candidate_term: 3 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" is_pre_election: true
I20260925 14:54:05.427017 27981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb753be525354422a7b3d2a3d659bcbc in term 2.
I20260925 14:54:05.427270 28144 leader_election.cc:302] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3d21a05a5941289d00c3aa6d0a28e9, bb753be525354422a7b3d2a3d659bcbc; no voters:
I20260925 14:54:05.427331 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.427236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50812)
0925 14:54:05.427284 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.427313 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:54:05.427381 28704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 14:54:05.427416 28704 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:05.427439 28704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 2 FOLLOWER]: Advancing to term 3
W20260925 14:54:05.427495 28144 leader_election.cc:334] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:05.428180 28704 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:05.428277 28704 leader_election.cc:288] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 3 election: Requested vote from peers 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765), 3f3d21a05a5941289d00c3aa6d0a28e9 (127.20.199.61:42661)
I20260925 14:54:05.428524 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.428450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50818)
0925 14:54:05.428481 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.428507 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 14:54:05.428615 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb753be525354422a7b3d2a3d659bcbc" candidate_term: 3 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9"
I20260925 14:54:05.428699 27981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:54:05.429391 27981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb753be525354422a7b3d2a3d659bcbc in term 3.
I20260925 14:54:05.429710 28144 leader_election.cc:302] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3d21a05a5941289d00c3aa6d0a28e9, bb753be525354422a7b3d2a3d659bcbc; no voters:
I20260925 14:54:05.429757 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.429673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50826)
0925 14:54:05.429700 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.429725 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 14:54:05.429826 28144 leader_election.cc:334] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
I20260925 14:54:05.429885 28704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:54:05.430029 28704 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [term 3 LEADER]: Becoming Leader. State: Replica: bb753be525354422a7b3d2a3d659bcbc, State: Running, Role: LEADER
I20260925 14:54:05.430114 28704 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } }
I20260925 14:54:05.430472 28707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb753be525354422a7b3d2a3d659bcbc. Latest consensus state: current_term: 3 leader_uuid: "bb753be525354422a7b3d2a3d659bcbc" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:05.430584 28707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:05.430785 28709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:05.431164 28709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:05.431224 28709 catalog_manager.cc:1259] Loaded cluster ID: a9e4316ad58a45548f6d7685232580fe
I20260925 14:54:05.431243 28709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:05.431411 28709 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:05.431474 28709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Loaded TSK: 0
I20260925 14:54:05.431576 28709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:05.499761 28149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:05.501556 28149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:05.502667 27981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [term 3 FOLLOWER]: Refusing update from remote peer bb753be525354422a7b3d2a3d659bcbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20260925 14:54:05.502729 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.502589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50830)
0925 14:54:05.502644 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.502674 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:54:05.502941 28707 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260925 14:54:05.503118 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.503002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.0.0.1:50846)
0925 14:54:05.503059 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.503091 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:54:05.503221 28144 consensus_peers.cc:487] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc -> Peer 684c9a750c1148949f3cd4cc17d297cb (127.20.199.62:40765): Couldn't send request to peer 684c9a750c1148949f3cd4cc17d297cb. Status: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:54:05.503535 28711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb753be525354422a7b3d2a3d659bcbc. Latest consensus state: current_term: 3 leader_uuid: "bb753be525354422a7b3d2a3d659bcbc" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:05.503644 28711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:05.504120 28707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bb753be525354422a7b3d2a3d659bcbc" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:05.504205 28707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:05.504338 28712 mvcc.cc:205] Tried to move back new op lower bound from 7333265594376941568 to 7333265594082463744. Current Snapshot: MvccSnapshot[applied={T|T < 7333265594376941568}]
I20260925 14:54:05.504451 28711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bb753be525354422a7b3d2a3d659bcbc" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "684c9a750c1148949f3cd4cc17d297cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40765 } } peers { permanent_uuid: "3f3d21a05a5941289d00c3aa6d0a28e9" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42661 } attrs { promote: false } } peers { permanent_uuid: "bb753be525354422a7b3d2a3d659bcbc" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 41853 } attrs { promote: false } } }
I20260925 14:54:05.504526 28711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3d21a05a5941289d00c3aa6d0a28e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:05.508960 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.505939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:44885 (local address 127.0.0.1:42356)
0925 14:54:05.506112 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.506123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:05.506173 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:05.506393 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:05.506393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:05.506395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:05.506491 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.506492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.507251 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.507252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.508070 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.508072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.508731 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.508732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.508792 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.508796 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:05.508848 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:54:05.508888 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":18,"thread_start_us":82,"threads_started":1}
I20260925 14:54:05.508967 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.506136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.20.199.3:44885)
0925 14:54:05.506257 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:05.506259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:05.506272 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:54:05.506295 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:05.506296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:05.506306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:05.506375 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:05.506564 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.507160 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.508157 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.508672 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.508734 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.508746 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:05.508860 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:54:05.508905 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260925 14:54:05.509344 28543 tablet_service.cc:1467] Processing CreateTablet for tablet 82abd6cc51664104b5eda73b83e10f47 (DEFAULT_TABLE table=test-table [id=2c573d878b6e44acb891bac0aeab99cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:05.509372 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.505894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:38101 (local address 127.0.0.1:51398)
0925 14:54:05.505938 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.505947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:05.506007 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:05.506389 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:05.506390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:05.506391 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:05.506467 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.506469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.507216 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.507217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.508405 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.508407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.509169 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.509170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.509230 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.509234 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:05.509265 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:54:05.509308 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":9}
I20260925 14:54:05.509518 28543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82abd6cc51664104b5eda73b83e10f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:05.509645 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.505959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.20.199.1:38101)
0925 14:54:05.506074 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:05.506077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:05.506086 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:05.506280 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:05.506281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:05.506294 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:05.506375 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:05.506558 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.507147 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.508517 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.509079 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.509509 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.509523 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:05.509541 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 14:54:05.509580 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260925 14:54:05.509851 28270 tablet_service.cc:1467] Processing CreateTablet for tablet 82abd6cc51664104b5eda73b83e10f47 (DEFAULT_TABLE table=test-table [id=2c573d878b6e44acb891bac0aeab99cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:05.509955 28270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82abd6cc51664104b5eda73b83e10f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:05.510538 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.506120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44423 (local address 127.0.0.1:45708)
0925 14:54:05.506440 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.506449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:05.506496 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:05.507365 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:05.507366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:05.507368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:05.507401 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.507402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.508289 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.508290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.509729 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.509732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.510275 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.510276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.510343 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.510346 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:05.510428 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:54:05.510465 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":65,"threads_started":1}
I20260925 14:54:05.510550 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.506189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.20.199.2:44423)
0925 14:54:05.506300 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:05.506302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:05.507251 (+ 949us) server_negotiation.cc:407] Connection header received
0925 14:54:05.507280 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:05.507281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:05.507293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:05.507351 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:05.507490 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.508193 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.509799 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.510194 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.510275 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.510286 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:05.510436 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:54:05.510483 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260925 14:54:05.510751 28407 tablet_service.cc:1467] Processing CreateTablet for tablet 82abd6cc51664104b5eda73b83e10f47 (DEFAULT_TABLE table=test-table [id=2c573d878b6e44acb891bac0aeab99cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:05.510850 28407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82abd6cc51664104b5eda73b83e10f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:05.511096 28722 tablet_bootstrap.cc:492] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Bootstrap starting.
I20260925 14:54:05.511507 28722 tablet_bootstrap.cc:654] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:05.512108 28723 tablet_bootstrap.cc:492] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: Bootstrap starting.
I20260925 14:54:05.512158 28722 tablet_bootstrap.cc:492] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: No bootstrap required, opened a new log
I20260925 14:54:05.512203 28722 ts_tablet_manager.cc:1397] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:05.512290 28725 tablet_bootstrap.cc:492] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Bootstrap starting.
I20260925 14:54:05.512503 28722 raft_consensus.cc:348] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.512530 28723 tablet_bootstrap.cc:654] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:05.512589 28722 raft_consensus.cc:374] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:05.512616 28722 raft_consensus.cc:729] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edec8472aec940b2995695543883ba11, State: Initialized, Role: FOLLOWER
I20260925 14:54:05.512614 28725 tablet_bootstrap.cc:654] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:05.512699 28722 consensus_queue.cc:260] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.512887 28722 ts_tablet_manager.cc:1428] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:05.513136 28723 tablet_bootstrap.cc:492] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: No bootstrap required, opened a new log
I20260925 14:54:05.513209 28723 ts_tablet_manager.cc:1397] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:05.513259 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.513140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:47483)
0925 14:54:05.513197 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.513232 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:54:05.513347 28723 raft_consensus.cc:348] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.513468 28723 raft_consensus.cc:374] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:05.513500 28723 raft_consensus.cc:729] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff97d18e016b4ca69b9e5f05af95ccf2, State: Initialized, Role: FOLLOWER
I20260925 14:54:05.513540 28723 consensus_queue.cc:260] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.513608 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.513495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:36225)
0925 14:54:05.513543 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.513572 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:54:05.513700 28723 ts_tablet_manager.cc:1428] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:05.514058 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.513921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:51535)
0925 14:54:05.513997 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.514030 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:54:05.514227 28725 tablet_bootstrap.cc:492] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: No bootstrap required, opened a new log
I20260925 14:54:05.514272 28725 ts_tablet_manager.cc:1397] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:05.514331 28336 heartbeater.cc:503] Master 127.20.199.60:41853 was elected leader, sending a full tablet report...
I20260925 14:54:05.514437 28725 raft_consensus.cc:348] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.514518 28725 raft_consensus.cc:374] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:05.514539 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.514436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:51379)
0925 14:54:05.514490 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.514512 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:54:05.514569 28725 raft_consensus.cc:729] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634f787031284d6f8d70941f91850b6d, State: Initialized, Role: FOLLOWER
I20260925 14:54:05.514619 28725 consensus_queue.cc:260] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.514827 28725 ts_tablet_manager.cc:1428] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:05.515121 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.515022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:53845)
0925 14:54:05.515061 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.515091 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:54:05.515328 28473 heartbeater.cc:503] Master 127.20.199.60:41853 was elected leader, sending a full tablet report...
I20260925 14:54:05.515522 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.515335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:44455)
0925 14:54:05.515472 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.515499 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 14:54:05.861907 28729 raft_consensus.cc:482] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:05.862037 28729 raft_consensus.cc:504] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.862247 28729 leader_election.cc:288] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edec8472aec940b2995695543883ba11 (127.20.199.3:44885), 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423)
I20260925 14:54:05.862475 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82abd6cc51664104b5eda73b83e10f47" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edec8472aec940b2995695543883ba11" is_pre_election: true
I20260925 14:54:05.862592 28563 raft_consensus.cc:2454] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 0.
I20260925 14:54:05.862543 28427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82abd6cc51664104b5eda73b83e10f47" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634f787031284d6f8d70941f91850b6d" is_pre_election: true
I20260925 14:54:05.862646 28427 raft_consensus.cc:2454] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 0.
I20260925 14:54:05.862771 28225 leader_election.cc:302] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edec8472aec940b2995695543883ba11, ff97d18e016b4ca69b9e5f05af95ccf2; no voters:
I20260925 14:54:05.862931 28729 raft_consensus.cc:2781] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:05.862972 28729 raft_consensus.cc:482] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:05.862986 28729 raft_consensus.cc:3037] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:05.863689 28729 raft_consensus.cc:504] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.863783 28729 leader_election.cc:288] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 election: Requested vote from peers edec8472aec940b2995695543883ba11 (127.20.199.3:44885), 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423)
I20260925 14:54:05.863938 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82abd6cc51664104b5eda73b83e10f47" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edec8472aec940b2995695543883ba11"
I20260925 14:54:05.863986 28427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82abd6cc51664104b5eda73b83e10f47" candidate_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634f787031284d6f8d70941f91850b6d"
I20260925 14:54:05.864030 28563 raft_consensus.cc:3037] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:05.864058 28427 raft_consensus.cc:3037] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:05.864795 28563 raft_consensus.cc:2454] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 1.
I20260925 14:54:05.864851 28427 raft_consensus.cc:2454] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff97d18e016b4ca69b9e5f05af95ccf2 in term 1.
I20260925 14:54:05.864977 28225 leader_election.cc:302] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edec8472aec940b2995695543883ba11, ff97d18e016b4ca69b9e5f05af95ccf2; no voters:
I20260925 14:54:05.865082 28729 raft_consensus.cc:2781] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:05.865224 28729 raft_consensus.cc:686] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 LEADER]: Becoming Leader. State: Replica: ff97d18e016b4ca69b9e5f05af95ccf2, State: Running, Role: LEADER
I20260925 14:54:05.865269 28729 consensus_queue.cc:237] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } }
I20260925 14:54:05.865694 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.865571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:38597)
0925 14:54:05.865627 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.865658 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:54:05.866168 28149 catalog_manager.cc:5495] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1). New cstate: current_term: 1 leader_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff97d18e016b4ca69b9e5f05af95ccf2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 38101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:05.949755 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.947481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.20.199.1:38101)
0925 14:54:05.947525 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:05.947527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:05.947544 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:54:05.947586 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:05.947587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:05.947605 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:05.947678 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:05.947790 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.948334 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.949044 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.949244 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.949302 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.949466 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:05.949568 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:05.949635 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:05.949673 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:54:05.949693 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:05.950974 28563 raft_consensus.cc:1261] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 1 FOLLOWER]: Refusing update from remote peer ff97d18e016b4ca69b9e5f05af95ccf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:05.951023 28427 raft_consensus.cc:1261] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Refusing update from remote peer ff97d18e016b4ca69b9e5f05af95ccf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:05.951259 28729 consensus_queue.cc:1035] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edec8472aec940b2995695543883ba11" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:05.951397 28731 consensus_queue.cc:1035] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634f787031284d6f8d70941f91850b6d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:05.952119 28609 heartbeater.cc:503] Master 127.20.199.60:41853 was elected leader, sending a full tablet report...
I20260925 14:54:05.952153 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.952023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:55471)
0925 14:54:05.952080 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.952113 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:54:05.952487 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.952345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:51137)
0925 14:54:05.952413 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.952458 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 14:54:05.954099 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.953970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.3:49275)
0925 14:54:05.954032 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.954069 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:54:05.954319 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.954223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.2:46473)
0925 14:54:05.954263 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.954286 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 14:54:05.954475 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.954368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:43315)
0925 14:54:05.954409 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.954444 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:54:05.954972 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.954861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40765 (local address 127.20.199.1:36933)
0925 14:54:05.954916 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.954943 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:54:05.957242 28148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:05.957335 28148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:05.958666 28148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb753be525354422a7b3d2a3d659bcbc: Sending DeleteTablet for 3 replicas of tablet 82abd6cc51664104b5eda73b83e10f47
I20260925 14:54:05.958964 28270 tablet_service.cc:1514] Processing DeleteTablet for tablet 82abd6cc51664104b5eda73b83e10f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:05 UTC) from {username='oryx', principal='oryx/127.20.199.60@KRBTEST.COM'} at 127.0.0.1:51398
I20260925 14:54:05.958988 28543 tablet_service.cc:1514] Processing DeleteTablet for tablet 82abd6cc51664104b5eda73b83e10f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:05 UTC) from {username='oryx', principal='oryx/127.20.199.60@KRBTEST.COM'} at 127.0.0.1:42356
I20260925 14:54:05.959008 28407 tablet_service.cc:1514] Processing DeleteTablet for tablet 82abd6cc51664104b5eda73b83e10f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:05 UTC) from {username='oryx', principal='oryx/127.20.199.60@KRBTEST.COM'} at 127.0.0.1:45708
I20260925 14:54:05.959190 28747 tablet_replica.cc:330] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: stopping tablet replica
I20260925 14:54:05.959241 28747 raft_consensus.cc:2229] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:05.959244 28746 tablet_replica.cc:330] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: stopping tablet replica
I20260925 14:54:05.959281 28747 raft_consensus.cc:2258] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:05.959298 28746 raft_consensus.cc:2229] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:05.959379 28746 raft_consensus.cc:2258] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:05.959378 28748 tablet_replica.cc:330] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: stopping tablet replica
I20260925 14:54:05.959429 28748 raft_consensus.cc:2229] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:05.959518 28748 raft_consensus.cc:2258] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:05.959694 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 28118
I20260925 14:54:05.959730 28746 ts_tablet_manager.cc:1905] T 82abd6cc51664104b5eda73b83e10f47 P ff97d18e016b4ca69b9e5f05af95ccf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:05.959846 28747 ts_tablet_manager.cc:1905] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:05.961372 28748 ts_tablet_manager.cc:1905] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:05.962646 28748 ts_tablet_manager.cc:1918] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:05.962692 28748 log.cc:1196] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/82abd6cc51664104b5eda73b83e10f47
I20260925 14:54:05.962917 28748 ts_tablet_manager.cc:1939] T 82abd6cc51664104b5eda73b83e10f47 P 634f787031284d6f8d70941f91850b6d: Deleting consensus metadata
I20260925 14:54:05.963039 28747 ts_tablet_manager.cc:1918] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:05.963097 28747 log.cc:1196] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/82abd6cc51664104b5eda73b83e10f47
I20260925 14:54:05.963304 28747 ts_tablet_manager.cc:1939] T 82abd6cc51664104b5eda73b83e10f47 P edec8472aec940b2995695543883ba11: Deleting consensus metadata
I20260925 14:54:05.963358 28143 catalog_manager.cc:4841] TS 634f787031284d6f8d70941f91850b6d (127.20.199.2:44423): tablet 82abd6cc51664104b5eda73b83e10f47 (table test-table [id=2c573d878b6e44acb891bac0aeab99cb]) successfully deleted
I20260925 14:54:05.963546 28143 catalog_manager.cc:4841] TS edec8472aec940b2995695543883ba11 (127.20.199.3:44885): tablet 82abd6cc51664104b5eda73b83e10f47 (table test-table [id=2c573d878b6e44acb891bac0aeab99cb]) successfully deleted
I20260925 14:54:05.966714 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.966510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:38101 (local address 127.0.0.1:51404)
0925 14:54:05.966591 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.966636 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:54:05.966770 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 28344
W20260925 14:54:05.967701 28142 catalog_manager.cc:4584] TS ff97d18e016b4ca69b9e5f05af95ccf2 (127.20.199.1:38101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82abd6cc51664104b5eda73b83e10f47: Network error: Client connection negotiation failed: client connection to 127.20.199.1:38101: connect: Connection refused (error 111)
I20260925 14:54:05.972759 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 28480
I20260925 14:54:05.977977 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 27838
I20260925 14:54:05.982726 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 28025
2026-09-25T14:54:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7096 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:06 dist-test-slave-8qt6 krb5kdc[28751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:06 dist-test-slave-8qt6 krb5kdc[28751](info): set up 2 sockets
Sep 25 14:54:06 dist-test-slave-8qt6 krb5kdc[28751](info): commencing operation
krb5kdc: starting...
W20260925 14:54:07.029877 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab.
Entry for principal oryx/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab.
I20260925 14:54:07.148692 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42923
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab
--principal=oryx/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:42923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:07.227810 28764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:07.228051 28764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:07.228087 28764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:07.229224 28764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:07.229295 28764 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:54:07.229317 28764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:07.229341 28764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:07.229370 28764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:07.229421 28764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:07.230952 28764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:42923
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.62.keytab
--principal=oryx/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:07.231227 28764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:07.231462 28764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:07.236246 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:07.236253 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:07.236390 28764 server_base.cc:1034] running on GCE node
W20260925 14:54:07.236253 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:07.236685 28764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:07.236963 28764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:07.238114 28764 hybrid_clock.cc:648] HybridClock initialized: now 1790348047238093 us; error 41 us; skew 500 ppm
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:07.240658 28764 init.cc:377] Logged in from keytab as oryx/127.20.199.62@KRBTEST.COM (short username oryx)
I20260925 14:54:07.241112 28764 webserver.cc:460] Webserver started at http://127.20.199.62:44473/ using document root <none> and password file <none>
I20260925 14:54:07.241360 28764 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:07.241456 28764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:07.241601 28764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:07.242651 28764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "6bdc0ee11469461fb325f5442f4e94cb"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "c293329ba7e169795fd814959d734602"
server_key_iv: "9f009134190f7a5cefae276b72e7953f"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.243002 28764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "6bdc0ee11469461fb325f5442f4e94cb"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "c293329ba7e169795fd814959d734602"
server_key_iv: "9f009134190f7a5cefae276b72e7953f"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.244292 28764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:07.245045 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.245213 28764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:07.245328 28764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "6bdc0ee11469461fb325f5442f4e94cb"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "c293329ba7e169795fd814959d734602"
server_key_iv: "9f009134190f7a5cefae276b72e7953f"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.245405 28764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:07.279805 28764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:07.282176 28764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:07.282361 28764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:07.287009 28764 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:42923
I20260925 14:54:07.287058 28841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:42923 every 8 connection(s)
I20260925 14:54:07.287386 28764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:07.287993 28842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.290349 28842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Bootstrap starting.
I20260925 14:54:07.290923 28842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.291214 28842 log.cc:824] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:07.291860 28842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: No bootstrap required, opened a new log
I20260925 14:54:07.293566 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 28764
I20260925 14:54:07.293638 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:54:07.293834 21276 external_mini_cluster.cc:1428] Setting key e8b918b18dcb435375f23ebfb7596c28
I20260925 14:54:07.294065 28842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdc0ee11469461fb325f5442f4e94cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42923 } }
I20260925 14:54:07.294222 28842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.294296 28842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdc0ee11469461fb325f5442f4e94cb, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.294516 28842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdc0ee11469461fb325f5442f4e94cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42923 } }
I20260925 14:54:07.294621 28842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:07.294648 28842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:07.294718 28842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.295483 28842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdc0ee11469461fb325f5442f4e94cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42923 } }
I20260925 14:54:07.295598 28842 leader_election.cc:302] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bdc0ee11469461fb325f5442f4e94cb; no voters:
I20260925 14:54:07.295763 28842 leader_election.cc:288] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:07.295802 28847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:07.295975 28847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [term 1 LEADER]: Becoming Leader. State: Replica: 6bdc0ee11469461fb325f5442f4e94cb, State: Running, Role: LEADER
I20260925 14:54:07.296068 28842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:07.296154 28847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdc0ee11469461fb325f5442f4e94cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42923 } }
I20260925 14:54:07.296551 28847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bdc0ee11469461fb325f5442f4e94cb. Latest consensus state: current_term: 1 leader_uuid: "6bdc0ee11469461fb325f5442f4e94cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdc0ee11469461fb325f5442f4e94cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42923 } } }
I20260925 14:54:07.296581 28849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bdc0ee11469461fb325f5442f4e94cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdc0ee11469461fb325f5442f4e94cb" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42923 } } }
I20260925 14:54:07.296703 28849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:07.296888 28847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:07.296891 28856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:07.297508 28856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:07.299257 28856 catalog_manager.cc:1347] Generated new cluster ID: fa26cc7948914cf5bd028e88f66a63e3
I20260925 14:54:07.299341 28856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:07.304914 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.294336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.20.199.62:42923)
0925 14:54:07.294509 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.294522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.294549 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:54:07.294677 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.294684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.294746 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.294892 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.295669 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.297092 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.298198 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.298767 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.298884 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.301436 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.301439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.301440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.301451 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.303333 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.303727 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.303730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.303730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.303838 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.304048 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.304048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.304048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.304329 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.304481 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.304609 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.304730 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260925 14:54:07.309427 28856 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:07.309947 28856 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:07.315562 28856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Generated new TSK 0
I20260925 14:54:07.315769 28856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab.
Entry for principal oryx/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab.
I20260925 14:54:07.350293 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42923
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:07.441341 28868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:07.441610 28868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:07.441645 28868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:07.442798 28868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:07.442836 28868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:07.442857 28868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:54:07.442873 28868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:07.444391 28868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:07.444646 28868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:07.444876 28868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:07.449870 28873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:07.449887 28876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:07.449954 28868 server_base.cc:1034] running on GCE node
W20260925 14:54:07.449885 28874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:07.450345 28868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:07.450584 28868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:07.451727 28868 hybrid_clock.cc:648] HybridClock initialized: now 1790348047451712 us; error 29 us; skew 500 ppm
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:07.454252 28868 init.cc:377] Logged in from keytab as oryx/127.20.199.1@KRBTEST.COM (short username oryx)
I20260925 14:54:07.454625 28868 webserver.cc:460] Webserver started at http://127.20.199.1:33913/ using document root <none> and password file <none>
I20260925 14:54:07.454820 28868 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:07.454860 28868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:07.454948 28868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:07.455938 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "71c20b836db747028fe52701eb0a75fb"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "bc0bb7632844eb314da9475b3d8a52af"
server_key_iv: "6620f56d05c195dda03402628de78676"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.456235 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "71c20b836db747028fe52701eb0a75fb"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "bc0bb7632844eb314da9475b3d8a52af"
server_key_iv: "6620f56d05c195dda03402628de78676"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.457515 28868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:54:07.458209 28883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.458412 28868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:07.458503 28868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "71c20b836db747028fe52701eb0a75fb"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "bc0bb7632844eb314da9475b3d8a52af"
server_key_iv: "6620f56d05c195dda03402628de78676"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.458562 28868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:07.468173 28868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:07.470459 28868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:07.470583 28868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:07.471006 28868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:07.471046 28868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.471096 28868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:07.471131 28868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.478101 28868 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:40731
I20260925 14:54:07.478206 28998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:40731 every 8 connection(s)
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:07.478547 28868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:07.482076 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.471528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:56091 (local address 127.20.199.62:42923)
0925 14:54:07.471600 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.471604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.471790 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:54:07.472281 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.472282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.472299 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.472387 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.473246 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.474106 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.475178 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.475504 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.475583 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.478947 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.478948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.478949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.478957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.480529 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.480987 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.480988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.480988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.481055 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.481325 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.481325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.481326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.481522 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.481613 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.481895 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.481982 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:07.482271 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.471484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.20.199.1:56091)
0925 14:54:07.471652 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.471693 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.472186 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.472455 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.472458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.472849 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.473132 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.473139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.474231 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.474232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.475041 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.475043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.475613 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.475615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.475795 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.476980 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.476992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.478835 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.480651 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.480658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.480660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.480887 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.481134 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.481135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.481136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.481249 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.481594 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.481601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.481742 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.481869 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.482024 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260925 14:54:07.482807 28999 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42923
I20260925 14:54:07.482911 28999 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:07.483081 28999 heartbeater.cc:511] Master 127.20.199.62:42923 requested a full tablet report, sending...
I20260925 14:54:07.483603 28795 ts_manager.cc:194] Registered new tserver with Master: 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731)
I20260925 14:54:07.484332 28795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:56091
I20260925 14:54:07.485690 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 28868
I20260925 14:54:07.485795 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:07.486006 21276 external_mini_cluster.cc:1428] Setting key 96219d49026ec11b67836d7117a07885
I20260925 14:54:07.490185 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.483919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58715 (local address 127.20.199.62:42923)
0925 14:54:07.483963 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.483966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.484038 (+ 72us) server_negotiation.cc:407] Connection header received
0925 14:54:07.484122 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.484123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.484133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.484190 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.484558 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.485026 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.485831 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.486085 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.486168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.487129 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.487130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.487131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.487138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.488952 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.489459 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.489464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.489465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.489518 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.489728 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.489728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.489728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.489832 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.489893 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.490066 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.490128 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:54:07.490324 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.483859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.20.199.1:58715)
0925 14:54:07.483982 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.483993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.484054 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.484195 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.484195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.484366 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.484458 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.484460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.485126 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.485126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.485714 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.485716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.486169 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.486170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.486241 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.486715 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.486720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.487028 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.489074 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.489075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.489075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.489333 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.489591 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.489592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.489592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.489664 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.489906 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.489907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.489968 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.490041 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.490164 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab.
Entry for principal oryx/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab.
I20260925 14:54:07.512632 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42923
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:07.592840 29008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:07.593035 29008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:07.593060 29008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:07.594328 29008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:07.594375 29008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:07.594398 29008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:54:07.594415 29008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:07.596029 29008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:07.596276 29008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:07.596550 29008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:07.601533 29016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:07.601588 29014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:07.601543 29013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:07.601784 29008 server_base.cc:1034] running on GCE node
I20260925 14:54:07.602036 29008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:07.602324 29008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:07.603521 29008 hybrid_clock.cc:648] HybridClock initialized: now 1790348047603502 us; error 44 us; skew 500 ppm
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:07.606201 29008 init.cc:377] Logged in from keytab as oryx/127.20.199.2@KRBTEST.COM (short username oryx)
I20260925 14:54:07.606662 29008 webserver.cc:460] Webserver started at http://127.20.199.2:40183/ using document root <none> and password file <none>
I20260925 14:54:07.606911 29008 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:07.606979 29008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:07.607125 29008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:07.608165 29008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "822ad9e7835a46b69310433699f075d4"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "9771bb5a660acf059fba394a8e53f19f"
server_key_iv: "1baa960b41a2c17dd7689dd0062dfc72"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.608532 29008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "822ad9e7835a46b69310433699f075d4"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "9771bb5a660acf059fba394a8e53f19f"
server_key_iv: "1baa960b41a2c17dd7689dd0062dfc72"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.609953 29008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:54:07.610656 29023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.610822 29008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:07.610949 29008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "822ad9e7835a46b69310433699f075d4"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "9771bb5a660acf059fba394a8e53f19f"
server_key_iv: "1baa960b41a2c17dd7689dd0062dfc72"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.611025 29008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:07.645673 29008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:07.647878 29008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:07.648041 29008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:07.648617 29008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:07.648681 29008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.648725 29008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:07.648753 29008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:07.656236 29008 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:46055
I20260925 14:54:07.656476 29138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:46055 every 8 connection(s)
I20260925 14:54:07.656662 29008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:07.657907 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29008
I20260925 14:54:07.658035 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:07.658316 21276 external_mini_cluster.cc:1428] Setting key bd5b91704c20e52fb5901360a479dbb5
I20260925 14:54:07.659256 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.649033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:38533 (local address 127.20.199.62:42923)
0925 14:54:07.649102 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.649105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.649368 (+ 263us) server_negotiation.cc:407] Connection header received
0925 14:54:07.649982 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.649984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.649998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.650080 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.650745 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.651584 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.652805 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.653143 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.653244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.655861 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.655863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.655864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.655873 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.657651 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.658182 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.658183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.658183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.658238 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.658524 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.658525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.658525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.658678 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.658729 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.659069 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.659153 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:07.659502 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.649077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.20.199.2:38533)
0925 14:54:07.649228 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.649272 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.649870 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.650154 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.650158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.650466 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.650664 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.650669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.651678 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.651679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.652694 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.652697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.653270 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.653271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.653468 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.654038 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.654050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.655754 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.657744 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.657750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.657752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.658099 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.658307 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.658308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.658308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.658455 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.658726 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.658733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.658897 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.659044 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.659215 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 14:54:07.659950 29139 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42923
I20260925 14:54:07.660034 29139 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:07.660257 29139 heartbeater.cc:511] Master 127.20.199.62:42923 requested a full tablet report, sending...
I20260925 14:54:07.660610 28796 ts_manager.cc:194] Registered new tserver with Master: 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055)
I20260925 14:54:07.661000 28796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:38533
WARNING: no policy specified for oryx/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:54:07.668192 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.661702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57875 (local address 127.20.199.62:42923)
0925 14:54:07.661756 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.661758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.661891 (+ 133us) server_negotiation.cc:407] Connection header received
0925 14:54:07.661953 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.661954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.661962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.662012 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.662434 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.662929 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.664053 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.664248 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.664315 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.665505 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.665506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.665507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.665514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.666970 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.667379 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.667381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.667381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.667445 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.667684 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.667685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.667685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.667816 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.667889 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.668070 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.668128 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:54:07.668196 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.661638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.20.199.2:57875)
0925 14:54:07.661819 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.661830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.661894 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.662017 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.662018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.662251 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.662330 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.662332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.663032 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.663033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.663950 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.663953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.664337 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.664338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.664422 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.664827 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.664832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.665319 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.667056 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.667057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.667058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.667277 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.667531 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.667532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.667533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.667606 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.667916 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.667917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.667973 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.668042 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.668121 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab.
Entry for principal oryx/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab.
I20260925 14:54:07.683607 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42923
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:07.764215 29148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:07.764444 29148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:07.764482 29148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:07.765733 29148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:07.765813 29148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:07.765849 29148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:54:07.765892 29148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:07.767547 29148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.199.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:07.767910 29148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:07.768184 29148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:07.773561 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:07.773563 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:07.773592 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:07.773758 29148 server_base.cc:1034] running on GCE node
I20260925 14:54:07.774140 29148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:07.774430 29148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:07.775607 29148 hybrid_clock.cc:648] HybridClock initialized: now 1790348047775582 us; error 48 us; skew 500 ppm
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:07.778344 29148 init.cc:377] Logged in from keytab as oryx/127.20.199.3@KRBTEST.COM (short username oryx)
I20260925 14:54:07.778820 29148 webserver.cc:460] Webserver started at http://127.20.199.3:38183/ using document root <none> and password file <none>
I20260925 14:54:07.779044 29148 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:07.779114 29148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:07.779268 29148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:07.780296 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "469d799f1ca6496e8bf621130ccdedc1"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "6710b1c05ec4adf414d3389489af7113"
server_key_iv: "4dd92a2ffe5e965bdcb3f8ee5a9100a4"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.780664 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "469d799f1ca6496e8bf621130ccdedc1"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "6710b1c05ec4adf414d3389489af7113"
server_key_iv: "4dd92a2ffe5e965bdcb3f8ee5a9100a4"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.782051 29148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:07.782797 29163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.782972 29148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:07.783107 29148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "469d799f1ca6496e8bf621130ccdedc1"
format_stamp: "Formatted at 2026-09-25 14:54:07 on dist-test-slave-8qt6"
server_key: "6710b1c05ec4adf414d3389489af7113"
server_key_iv: "4dd92a2ffe5e965bdcb3f8ee5a9100a4"
server_key_version: "encryptionkey@0"
I20260925 14:54:07.783190 29148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:07.802369 29148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:07.804598 29148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:07.804767 29148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:07.805398 29148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:07.805472 29148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.805511 29148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:07.805540 29148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:07.813375 29148 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:40953
I20260925 14:54:07.813476 29278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:40953 every 8 connection(s)
I20260925 14:54:07.813882 29148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:07.815308 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.805782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:36473 (local address 127.20.199.62:42923)
0925 14:54:07.805821 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.805823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.806053 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:54:07.806648 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.806650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.806665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.806728 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.807437 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.808052 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.809090 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.809292 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.809378 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.812005 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.812007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.812008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.812016 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.813674 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.814150 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.814151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.814152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.814227 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.814638 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.814639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.814639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.814765 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.814842 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.815139 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.815217 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:07.815505 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.805804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.20.199.3:36473)
0925 14:54:07.805938 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.805973 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.806498 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.806806 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.806809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.807160 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.807344 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.807349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.808157 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.808158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.808982 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.808984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.809388 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.809389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.809585 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.810108 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.810116 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.811914 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.813784 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.813790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.813792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.814064 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.814420 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.814421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.814422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.814556 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.814833 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.814839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.814990 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.815113 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.815269 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260925 14:54:07.816048 29279 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42923
I20260925 14:54:07.816164 29279 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:07.816341 29279 heartbeater.cc:511] Master 127.20.199.62:42923 requested a full tablet report, sending...
I20260925 14:54:07.816763 28796 ts_manager.cc:194] Registered new tserver with Master: 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:07.817229 28796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:36473
I20260925 14:54:07.818125 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29148
I20260925 14:54:07.818231 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:07.818410 21276 external_mini_cluster.cc:1428] Setting key 4d3a9bea74ee87de3ef912bea3855b39
I20260925 14:54:07.819271 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:07.823020 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.817177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:52037 (local address 127.20.199.62:42923)
0925 14:54:07.817221 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.817223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.817251 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:54:07.817319 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.817321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.817332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.817431 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.817772 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.818499 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.819345 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.819563 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.819654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.820538 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.820540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.820540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.820547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.821909 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.822260 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.822261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.822261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.822311 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.822518 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.822519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.822519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.822625 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.822698 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.822850 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.822923 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:07.823030 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.817068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.20.199.3:52037)
0925 14:54:07.817195 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.817206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.817266 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.817427 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.817428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.817635 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.817695 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.817697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.818596 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.818597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.819252 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.819254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.819657 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.819658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.819721 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.820163 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.820168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.820434 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.822010 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.822011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.822012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.822179 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.822389 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.822390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.822390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.822452 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.822691 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.822692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.822736 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.822824 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.822925 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:54:07.840420 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:07.834965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.20.199.62:42923)
0925 14:54:07.835027 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.835029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.835090 (+ 61us) server_negotiation.cc:407] Connection header received
0925 14:54:07.835179 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.835180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.835190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.835270 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.835856 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.836340 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.837121 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.837294 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.837368 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.840330 (+ 2962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48294: BlockingRecv error: recv got EOF from 127.0.0.1:48294 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:07.848387 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.841162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.20.199.62:42923)
0925 14:54:07.841225 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.841227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.841303 (+ 76us) server_negotiation.cc:407] Connection header received
0925 14:54:07.841421 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.841422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.841432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.841497 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.842007 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.842531 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.843278 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.843486 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.843576 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.845851 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.845853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.845853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.845860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.847148 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.847554 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.847555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.847556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.847620 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.847848 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.847849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.847849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.847965 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.848087 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.848199 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.848279 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:54:07.850117 28796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:07.851641 28796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:07.858530 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.854304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.20.199.1:40731)
0925 14:54:07.854481 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.854486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.854568 (+ 82us) server_negotiation.cc:407] Connection header received
0925 14:54:07.854632 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.854634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.854651 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.854905 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.854966 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.855787 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.857223 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.858051 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.858134 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.858216 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.858366 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.858411 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260925 14:54:07.858516 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.854005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:46055 (local address 127.0.0.1:54058)
0925 14:54:07.854211 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.854256 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.854333 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.854735 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.854739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.854742 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.854828 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.854830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.856313 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.856314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.857475 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.857478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.858223 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.858224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.858277 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.858279 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.858312 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.858373 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":8,"thread_start_us":64,"threads_started":1}
I20260925 14:54:07.858516 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.854024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.0.0.1:42418)
0925 14:54:07.854446 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.854459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.854543 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.854760 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.854761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.854763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.854817 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.854819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.855876 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.855877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.857133 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.857135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.858152 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.858153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.858245 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.858261 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.858340 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.858379 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260925 14:54:07.858868 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.854021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:40953 (local address 127.0.0.1:59266)
0925 14:54:07.854309 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.854320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.854365 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.854996 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.854997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.854999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.855037 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.855038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.856375 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.856376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.857506 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.857509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.858601 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.858603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.858686 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.858691 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.858753 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.858804 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":136,"threads_started":1}
I20260925 14:54:07.859050 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.854125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.20.199.2:46055)
0925 14:54:07.854490 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.854494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.854503 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:07.854535 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.854536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.854550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.854922 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.855000 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.856217 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.857571 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.858095 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.858766 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.858847 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.858895 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.858946 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":88,"threads_started":1}
I20260925 14:54:07.859127 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.854276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.20.199.3:40953)
0925 14:54:07.854688 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.854692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.854702 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:07.854823 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.854825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.854845 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.854921 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.855136 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.856275 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.857604 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.858411 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.858483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.858559 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.858963 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.859002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":88,"threads_started":1}
I20260925 14:54:07.859695 29073 tablet_service.cc:1467] Processing CreateTablet for tablet 861ef0dd0aa9442aacdbd6147be52808 (DEFAULT_TABLE table=test-table [id=a55a404bbd654f598059f937ef9b935e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:07.859787 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 861ef0dd0aa9442aacdbd6147be52808 (DEFAULT_TABLE table=test-table [id=a55a404bbd654f598059f937ef9b935e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:07.860086 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ef0dd0aa9442aacdbd6147be52808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.860174 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ef0dd0aa9442aacdbd6147be52808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.860170 28933 tablet_service.cc:1467] Processing CreateTablet for tablet 861ef0dd0aa9442aacdbd6147be52808 (DEFAULT_TABLE table=test-table [id=a55a404bbd654f598059f937ef9b935e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:07.860513 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ef0dd0aa9442aacdbd6147be52808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.863466 29307 tablet_bootstrap.cc:492] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Bootstrap starting.
I20260925 14:54:07.864104 29307 tablet_bootstrap.cc:654] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.864449 29308 tablet_bootstrap.cc:492] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Bootstrap starting.
I20260925 14:54:07.864475 29307 log.cc:824] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:07.864804 29309 tablet_bootstrap.cc:492] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Bootstrap starting.
I20260925 14:54:07.865120 29308 tablet_bootstrap.cc:654] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.865489 29309 tablet_bootstrap.cc:654] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.865527 29308 log.cc:824] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:07.865571 29307 tablet_bootstrap.cc:492] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: No bootstrap required, opened a new log
I20260925 14:54:07.865659 29307 ts_tablet_manager.cc:1397] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:07.865840 29309 log.cc:824] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:07.866654 29309 tablet_bootstrap.cc:492] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: No bootstrap required, opened a new log
I20260925 14:54:07.866652 29308 tablet_bootstrap.cc:492] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: No bootstrap required, opened a new log
I20260925 14:54:07.866729 29309 ts_tablet_manager.cc:1397] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:54:07.866729 29308 ts_tablet_manager.cc:1397] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:54:07.867511 29307 raft_consensus.cc:348] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.867676 29307 raft_consensus.cc:374] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.867722 29307 raft_consensus.cc:729] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469d799f1ca6496e8bf621130ccdedc1, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.867858 29307 consensus_queue.cc:260] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.868135 29307 ts_tablet_manager.cc:1428] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:54:07.868151 29279 heartbeater.cc:503] Master 127.20.199.62:42923 was elected leader, sending a full tablet report...
I20260925 14:54:07.868913 29308 raft_consensus.cc:348] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.868935 29309 raft_consensus.cc:348] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.869091 29309 raft_consensus.cc:374] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.869091 29308 raft_consensus.cc:374] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.869234 29309 raft_consensus.cc:729] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.869292 29308 raft_consensus.cc:729] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.869459 29308 consensus_queue.cc:260] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.869393 29309 consensus_queue.cc:260] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.869764 29308 ts_tablet_manager.cc:1428] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:54:07.869797 29309 ts_tablet_manager.cc:1428] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:54:07.869824 29139 heartbeater.cc:503] Master 127.20.199.62:42923 was elected leader, sending a full tablet report...
I20260925 14:54:07.869925 28999 heartbeater.cc:503] Master 127.20.199.62:42923 was elected leader, sending a full tablet report...
I20260925 14:54:07.885716 29315 raft_consensus.cc:482] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:07.885819 29315 raft_consensus.cc:504] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.886204 29315 leader_election.cc:288] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953), 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731)
I20260925 14:54:07.889232 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.886399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:52609 (local address 127.20.199.3:40953)
0925 14:54:07.886447 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.886449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.886459 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:07.886533 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.886534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.886550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.886631 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.886800 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.887364 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.888447 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.888817 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.888870 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.888877 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.889112 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.889149 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:07.889246 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.886350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:40953 (local address 127.20.199.2:52609)
0925 14:54:07.886404 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.886416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.886459 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.886640 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.886641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.886643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.886726 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.886729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.887436 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.887437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.888359 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.888361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.888892 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.888894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.889003 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.889007 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.889062 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.889140 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":19}
I20260925 14:54:07.889339 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.886448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:39019 (local address 127.20.199.1:40731)
0925 14:54:07.886486 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.886487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.886705 (+ 218us) server_negotiation.cc:407] Connection header received
0925 14:54:07.886763 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.886764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.886776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.886842 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.886996 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.887571 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.888519 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.888934 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.889009 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.889018 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.889227 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.889271 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:07.889601 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.886357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.20.199.2:39019)
0925 14:54:07.886640 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.886650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.886703 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.886845 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.886846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.886847 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.886909 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.886911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.887667 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.887668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.888431 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.888434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.889050 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.889051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.889133 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.889137 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.889439 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.889488 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":107,"thread_start_us":133,"threads_started":1}
I20260925 14:54:07.889599 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ef0dd0aa9442aacdbd6147be52808" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1" is_pre_election: true
I20260925 14:54:07.889791 29233 raft_consensus.cc:2454] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 0.
I20260925 14:54:07.890038 29025 leader_election.cc:302] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469d799f1ca6496e8bf621130ccdedc1, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:07.890033 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ef0dd0aa9442aacdbd6147be52808" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c20b836db747028fe52701eb0a75fb" is_pre_election: true
I20260925 14:54:07.890172 28953 raft_consensus.cc:2454] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 0.
I20260925 14:54:07.890230 29315 raft_consensus.cc:2781] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:07.890311 29315 raft_consensus.cc:482] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:07.890340 29315 raft_consensus.cc:3037] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.891295 29315 raft_consensus.cc:504] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.891434 29315 leader_election.cc:288] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 election: Requested vote from peers 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953), 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731)
I20260925 14:54:07.891582 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ef0dd0aa9442aacdbd6147be52808" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c20b836db747028fe52701eb0a75fb"
I20260925 14:54:07.891661 28953 raft_consensus.cc:3037] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.892367 28953 raft_consensus.cc:2454] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 1.
I20260925 14:54:07.892478 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ef0dd0aa9442aacdbd6147be52808" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1"
I20260925 14:54:07.892539 29026 leader_election.cc:302] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c20b836db747028fe52701eb0a75fb, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:07.892555 29233 raft_consensus.cc:3037] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.892658 29315 raft_consensus.cc:2781] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:07.892802 29315 raft_consensus.cc:686] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 1 LEADER]: Becoming Leader. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Running, Role: LEADER
I20260925 14:54:07.892910 29315 consensus_queue.cc:237] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } }
I20260925 14:54:07.893481 29233 raft_consensus.cc:2454] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 1.
I20260925 14:54:07.893829 28795 catalog_manager.cc:5495] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 822ad9e7835a46b69310433699f075d4 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "822ad9e7835a46b69310433699f075d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:07.903272 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.900760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.20.199.2:46055)
0925 14:54:07.900824 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.900826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.900835 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:07.900967 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.900969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.900988 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.901058 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:07.901253 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.901782 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.902604 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.902783 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.902832 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.902983 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:07.903065 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:07.903121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.903169 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.903185 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:54:07.904944 28953 raft_consensus.cc:1261] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Refusing update from remote peer 822ad9e7835a46b69310433699f075d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:07.904964 29233 raft_consensus.cc:1261] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Refusing update from remote peer 822ad9e7835a46b69310433699f075d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:07.905308 29315 consensus_queue.cc:1035] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:07.905471 29317 consensus_queue.cc:1035] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:54:07.907203 29140 tablet.cc:2369] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:07.907845 29320 mvcc.cc:205] Tried to move back new op lower bound from 7333265604216143872 to 7333265604170199040. Current Snapshot: MvccSnapshot[applied={T|T < 7333265604216143872}]
I20260925 14:54:07.908236 29321 mvcc.cc:205] Tried to move back new op lower bound from 7333265604216143872 to 7333265604170199040. Current Snapshot: MvccSnapshot[applied={T|T < 7333265604216143872}]
I20260925 14:54:07.910787 28796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:07.910879 28796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:07.911617 28796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Sending DeleteTablet for 3 replicas of tablet 861ef0dd0aa9442aacdbd6147be52808
I20260925 14:54:07.911952 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 861ef0dd0aa9442aacdbd6147be52808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:07 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:59266
I20260925 14:54:07.912160 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet 861ef0dd0aa9442aacdbd6147be52808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:07 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260925 14:54:07.912168 29330 tablet_replica.cc:330] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: stopping tablet replica
I20260925 14:54:07.912293 29330 raft_consensus.cc:2229] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:07.912405 29330 raft_consensus.cc:2258] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:07.912470 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet 861ef0dd0aa9442aacdbd6147be52808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:07 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:42418
I20260925 14:54:07.912577 29331 tablet_replica.cc:330] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: stopping tablet replica
I20260925 14:54:07.912658 29331 raft_consensus.cc:2229] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:07.912665 28796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:07.912807 28796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:07.912851 29331 raft_consensus.cc:2258] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:07.913362 29330 ts_tablet_manager.cc:1905] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:07.914335 28933 tablet_service.cc:1467] Processing CreateTablet for tablet 167843c8971a45b7982e5717e1c9c7f3 (DEFAULT_TABLE table=test-table [id=eb5ba981e5764831b6ca2c31d8945500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:07.914353 29073 tablet_service.cc:1467] Processing CreateTablet for tablet 167843c8971a45b7982e5717e1c9c7f3 (DEFAULT_TABLE table=test-table [id=eb5ba981e5764831b6ca2c31d8945500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:07.914458 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167843c8971a45b7982e5717e1c9c7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.914462 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167843c8971a45b7982e5717e1c9c7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.915391 29332 tablet_replica.cc:330] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: stopping tablet replica
I20260925 14:54:07.915490 29332 raft_consensus.cc:2229] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:07.915592 29332 raft_consensus.cc:2258] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:07.915748 29330 ts_tablet_manager.cc:1918] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:07.915807 29330 log.cc:1196] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/861ef0dd0aa9442aacdbd6147be52808
I20260925 14:54:07.915885 29308 tablet_bootstrap.cc:492] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Bootstrap starting.
I20260925 14:54:07.916059 29330 ts_tablet_manager.cc:1939] T 861ef0dd0aa9442aacdbd6147be52808 P 469d799f1ca6496e8bf621130ccdedc1: Deleting consensus metadata
I20260925 14:54:07.916111 29309 tablet_bootstrap.cc:492] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Bootstrap starting.
I20260925 14:54:07.916307 29308 tablet_bootstrap.cc:654] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.916533 29309 tablet_bootstrap.cc:654] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.916848 28781 catalog_manager.cc:4841] TS 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953): tablet 861ef0dd0aa9442aacdbd6147be52808 (table test-table [id=a55a404bbd654f598059f937ef9b935e]) successfully deleted
I20260925 14:54:07.916929 29308 tablet_bootstrap.cc:492] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: No bootstrap required, opened a new log
I20260925 14:54:07.916941 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 167843c8971a45b7982e5717e1c9c7f3 (DEFAULT_TABLE table=test-table [id=eb5ba981e5764831b6ca2c31d8945500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:07.916973 29308 ts_tablet_manager.cc:1397] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:07.917032 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167843c8971a45b7982e5717e1c9c7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.917114 29308 raft_consensus.cc:348] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.917203 29308 raft_consensus.cc:374] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.917225 29308 raft_consensus.cc:729] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.917263 29308 consensus_queue.cc:260] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.917340 29308 ts_tablet_manager.cc:1428] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.917435 29331 ts_tablet_manager.cc:1905] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:07.918286 29307 tablet_bootstrap.cc:492] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Bootstrap starting.
I20260925 14:54:07.918570 29332 ts_tablet_manager.cc:1905] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:07.918677 29307 tablet_bootstrap.cc:654] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.918998 29331 ts_tablet_manager.cc:1918] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:07.919070 29331 log.cc:1196] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/861ef0dd0aa9442aacdbd6147be52808
I20260925 14:54:07.919327 29331 ts_tablet_manager.cc:1939] T 861ef0dd0aa9442aacdbd6147be52808 P 822ad9e7835a46b69310433699f075d4: Deleting consensus metadata
I20260925 14:54:07.919721 28783 catalog_manager.cc:4841] TS 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055): tablet 861ef0dd0aa9442aacdbd6147be52808 (table test-table [id=a55a404bbd654f598059f937ef9b935e]) successfully deleted
I20260925 14:54:07.920106 29332 ts_tablet_manager.cc:1918] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:07.920198 29332 log.cc:1196] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/861ef0dd0aa9442aacdbd6147be52808
I20260925 14:54:07.920413 29332 ts_tablet_manager.cc:1939] T 861ef0dd0aa9442aacdbd6147be52808 P 71c20b836db747028fe52701eb0a75fb: Deleting consensus metadata
I20260925 14:54:07.920845 29307 tablet_bootstrap.cc:492] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: No bootstrap required, opened a new log
I20260925 14:54:07.920892 29307 ts_tablet_manager.cc:1397] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:54:07.920894 28782 catalog_manager.cc:4841] TS 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731): tablet 861ef0dd0aa9442aacdbd6147be52808 (table test-table [id=a55a404bbd654f598059f937ef9b935e]) successfully deleted
I20260925 14:54:07.921031 29307 raft_consensus.cc:348] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.921119 29307 raft_consensus.cc:374] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.921142 29307 raft_consensus.cc:729] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469d799f1ca6496e8bf621130ccdedc1, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.921198 29307 consensus_queue.cc:260] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.921293 29307 ts_tablet_manager.cc:1428] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:07.921749 29309 tablet_bootstrap.cc:492] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: No bootstrap required, opened a new log
I20260925 14:54:07.921813 29309 ts_tablet_manager.cc:1397] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20260925 14:54:07.921960 29309 raft_consensus.cc:348] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.922030 29309 raft_consensus.cc:374] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.922063 29309 raft_consensus.cc:729] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.922113 29309 consensus_queue.cc:260] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.922204 29309 ts_tablet_manager.cc:1428] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.943414 29314 raft_consensus.cc:482] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:07.943516 29314 raft_consensus.cc:504] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.943917 29314 leader_election.cc:288] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055), 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:07.947675 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.944290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:43117 (local address 127.20.199.3:40953)
0925 14:54:07.944333 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.944335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.944396 (+ 61us) server_negotiation.cc:407] Connection header received
0925 14:54:07.944475 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.944476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.944485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.944542 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.944705 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.945161 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.946711 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.947295 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.947376 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.947386 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.947559 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.947606 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:07.947708 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.944111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:40953 (local address 127.20.199.1:43117)
0925 14:54:07.944346 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.944355 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.944408 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.944546 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.944547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.944549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.944633 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.944635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.945250 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.945251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.946618 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.946620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.947390 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.947391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.947488 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.947494 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.947552 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.947597 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":60,"thread_start_us":60,"threads_started":1}
I20260925 14:54:07.947932 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167843c8971a45b7982e5717e1c9c7f3" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1" is_pre_election: true
I20260925 14:54:07.948019 29233 raft_consensus.cc:2454] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 0.
I20260925 14:54:07.948202 28885 leader_election.cc:302] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469d799f1ca6496e8bf621130ccdedc1, 71c20b836db747028fe52701eb0a75fb; no voters:
I20260925 14:54:07.948333 29314 raft_consensus.cc:2781] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:07.948369 29314 raft_consensus.cc:482] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:07.948407 29314 raft_consensus.cc:3037] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.948506 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.944119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:42545 (local address 127.20.199.2:46055)
0925 14:54:07.944163 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.944165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.944176 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:07.944246 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.944247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.944258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.944311 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:07.944541 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.945288 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.947426 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.948016 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.948081 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.948097 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.948402 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.948442 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:54:07.948678 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.944064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:46055 (local address 127.20.199.1:42545)
0925 14:54:07.944125 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.944138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.944176 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.944364 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.944365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.944366 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:07.944453 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.944455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.945367 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.945367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.946618 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.946620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.948150 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.948151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.948244 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.948248 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.948298 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.948339 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":4}
I20260925 14:54:07.948944 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167843c8971a45b7982e5717e1c9c7f3" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ad9e7835a46b69310433699f075d4" is_pre_election: true
I20260925 14:54:07.949095 29093 raft_consensus.cc:2454] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 0.
I20260925 14:54:07.949357 29314 raft_consensus.cc:504] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.949546 29314 leader_election.cc:288] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 election: Requested vote from peers 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055), 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:07.949705 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167843c8971a45b7982e5717e1c9c7f3" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1"
I20260925 14:54:07.949730 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167843c8971a45b7982e5717e1c9c7f3" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ad9e7835a46b69310433699f075d4"
I20260925 14:54:07.949783 29233 raft_consensus.cc:3037] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.949812 29093 raft_consensus.cc:3037] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:07.950420 29093 raft_consensus.cc:2454] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 1.
I20260925 14:54:07.950544 29233 raft_consensus.cc:2454] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 1.
I20260925 14:54:07.950591 28887 leader_election.cc:302] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c20b836db747028fe52701eb0a75fb, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:07.950693 29314 raft_consensus.cc:2781] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:07.950835 29314 raft_consensus.cc:686] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 1 LEADER]: Becoming Leader. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Running, Role: LEADER
I20260925 14:54:07.950911 29314 consensus_queue.cc:237] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.951633 28796 catalog_manager.cc:5495] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c20b836db747028fe52701eb0a75fb (127.20.199.1). New cstate: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } health_report { overall_health: UNKNOWN } } }
W20260925 14:54:07.979676 29000 tablet.cc:2369] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:54:07 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.62@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:07.989118 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.982159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.20.199.62:42923)
0925 14:54:07.982234 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:07.982237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:07.982371 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:54:07.982469 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:07.982470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:07.982480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:07.982535 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:07.982962 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.983513 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.984189 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.984381 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.984451 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.986632 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:07.986634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:07.986634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:07.986641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:07.987962 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.988323 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.988325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.988325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.988381 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:07.988570 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:07.988571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:07.988571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:07.988679 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:07.988760 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:07.988911 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:54:07.988992 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:54:07.989190 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.982104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42923 (local address 127.0.0.1:48312)
0925 14:54:07.982294 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.982302 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:07.982403 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:07.982547 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:07.982548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:07.982832 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:07.982876 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.982878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.983584 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.983585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.984101 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:07.984102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:07.984448 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:07.984449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:07.984499 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:07.985072 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:07.985077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:07.986475 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:07.988041 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.988048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.988050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.988248 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.988446 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:07.988447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:07.988447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:07.988504 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:07.988750 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:07.988757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:07.988801 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:54:07.988887 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:54:07.989046 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":64,"thread_start_us":98,"threads_started":1}
I20260925 14:54:07.991261 28796 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:48312:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:54:07.991452 28796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:07.992801 28933 tablet_service.cc:1467] Processing CreateTablet for tablet a5943fee3fbe4093aa8576bd3698ddfc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8acd9d9c1a2e4ea5993d79f5631b3134]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:54:07.992822 29073 tablet_service.cc:1467] Processing CreateTablet for tablet a5943fee3fbe4093aa8576bd3698ddfc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8acd9d9c1a2e4ea5993d79f5631b3134]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:54:07.992954 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5943fee3fbe4093aa8576bd3698ddfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.992954 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5943fee3fbe4093aa8576bd3698ddfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.992904 29213 tablet_service.cc:1467] Processing CreateTablet for tablet a5943fee3fbe4093aa8576bd3698ddfc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8acd9d9c1a2e4ea5993d79f5631b3134]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:54:07.993062 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5943fee3fbe4093aa8576bd3698ddfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:07.994416 29308 tablet_bootstrap.cc:492] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: Bootstrap starting.
I20260925 14:54:07.994793 29307 tablet_bootstrap.cc:492] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: Bootstrap starting.
I20260925 14:54:07.995105 29308 tablet_bootstrap.cc:654] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.995226 29307 tablet_bootstrap.cc:654] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.995479 29309 tablet_bootstrap.cc:492] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: Bootstrap starting.
I20260925 14:54:07.995857 29309 tablet_bootstrap.cc:654] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:07.995905 29307 tablet_bootstrap.cc:492] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: No bootstrap required, opened a new log
I20260925 14:54:07.995951 29307 ts_tablet_manager.cc:1397] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:07.996097 29307 raft_consensus.cc:348] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.996178 29307 raft_consensus.cc:374] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.996201 29307 raft_consensus.cc:729] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469d799f1ca6496e8bf621130ccdedc1, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.996260 29307 consensus_queue.cc:260] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.996373 29307 ts_tablet_manager.cc:1428] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:07.996539 29313 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:07.996601 29308 tablet_bootstrap.cc:492] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: No bootstrap required, opened a new log
I20260925 14:54:07.996622 29313 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:07.996644 29308 ts_tablet_manager.cc:1397] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:54:07.996665 29309 tablet_bootstrap.cc:492] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: No bootstrap required, opened a new log
I20260925 14:54:07.996702 29309 ts_tablet_manager.cc:1397] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:07.996833 29309 raft_consensus.cc:348] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.996821 29308 raft_consensus.cc:348] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.996889 29309 raft_consensus.cc:374] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.996894 29308 raft_consensus.cc:374] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:07.996912 29309 raft_consensus.cc:729] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.996917 29308 raft_consensus.cc:729] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Initialized, Role: FOLLOWER
I20260925 14:54:07.996961 29309 consensus_queue.cc:260] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.996964 29308 consensus_queue.cc:260] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:07.997042 29308 ts_tablet_manager.cc:1428] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:07.997044 29309 ts_tablet_manager.cc:1428] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:07.997136 29315 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:07.997206 29315 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:07.997251 29314 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:07.997340 29314 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:08.056277 29314 raft_consensus.cc:482] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:08.056372 29314 raft_consensus.cc:504] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.056556 29314 leader_election.cc:288] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055), 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:08.056758 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5943fee3fbe4093aa8576bd3698ddfc" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ad9e7835a46b69310433699f075d4" is_pre_election: true
I20260925 14:54:08.056859 29093 raft_consensus.cc:2454] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 0.
I20260925 14:54:08.056819 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5943fee3fbe4093aa8576bd3698ddfc" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1" is_pre_election: true
I20260925 14:54:08.056960 29233 raft_consensus.cc:2454] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 0.
I20260925 14:54:08.057015 28887 leader_election.cc:302] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c20b836db747028fe52701eb0a75fb, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:08.057133 29314 raft_consensus.cc:2781] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:08.057188 29314 raft_consensus.cc:482] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:08.057211 29314 raft_consensus.cc:3037] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.057888 29314 raft_consensus.cc:504] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.058027 29314 leader_election.cc:288] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 election: Requested vote from peers 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055), 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:08.058182 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5943fee3fbe4093aa8576bd3698ddfc" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1"
I20260925 14:54:08.058195 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5943fee3fbe4093aa8576bd3698ddfc" candidate_uuid: "71c20b836db747028fe52701eb0a75fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ad9e7835a46b69310433699f075d4"
I20260925 14:54:08.058262 29233 raft_consensus.cc:3037] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.058267 29093 raft_consensus.cc:3037] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.058985 29093 raft_consensus.cc:2454] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 1.
I20260925 14:54:08.058985 29233 raft_consensus.cc:2454] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c20b836db747028fe52701eb0a75fb in term 1.
I20260925 14:54:08.059165 28885 leader_election.cc:302] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469d799f1ca6496e8bf621130ccdedc1, 71c20b836db747028fe52701eb0a75fb; no voters:
I20260925 14:54:08.059294 29314 raft_consensus.cc:2781] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:08.059345 29314 raft_consensus.cc:686] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [term 1 LEADER]: Becoming Leader. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Running, Role: LEADER
I20260925 14:54:08.059384 29314 consensus_queue.cc:237] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.059686 29337 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: TxnStatusTablet state changed. Reason: New leader 71c20b836db747028fe52701eb0a75fb. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.059774 29337 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:54:08.059923 29349 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:54:08.060041 29349 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:54:08.059973 28796 catalog_manager.cc:5495] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c20b836db747028fe52701eb0a75fb (127.20.199.1). New cstate: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } health_report { overall_health: UNKNOWN } } }
W20260925 14:54:08.064535 29280 tablet.cc:2369] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:08.074664 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.071961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.20.199.1:40731)
0925 14:54:08.072017 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.072019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.072028 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:08.072108 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.072109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.072124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.072182 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.072330 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.072800 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.073926 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.074111 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.074180 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.074372 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.074458 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.074512 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.074594 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.074612 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:54:08.074715 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.071902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.0.0.1:42420)
0925 14:54:08.071980 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.071991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:08.072031 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:08.072204 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:08.072205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:08.072206 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:54:08.072268 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.072271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.072870 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.072871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.073826 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.073829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.074193 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.074194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.074266 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.074282 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.074520 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.074528 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:54:08.074578 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:54:08.074624 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:54:08.075699 29233 raft_consensus.cc:1261] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Refusing update from remote peer 71c20b836db747028fe52701eb0a75fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.075769 29093 raft_consensus.cc:1261] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Refusing update from remote peer 71c20b836db747028fe52701eb0a75fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.075939 29314 consensus_queue.cc:1035] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.076031 29337 consensus_queue.cc:1035] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.076722 29313 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: TxnStatusTablet state changed. Reason: New leader 71c20b836db747028fe52701eb0a75fb. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.076759 29315 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: TxnStatusTablet state changed. Reason: New leader 71c20b836db747028fe52701eb0a75fb. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.076829 29313 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:08.076839 29315 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:08.077637 29314 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.077718 29314 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:54:08.077764 29314 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.077814 29314 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 71c20b836db747028fe52701eb0a75fb: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:54:08.078235 29322 mvcc.cc:205] Tried to move back new op lower bound from 7333265604916293632 to 7333265604851650560. Current Snapshot: MvccSnapshot[applied={T|T < 7333265604916293632}]
I20260925 14:54:08.078336 29313 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.078397 29313 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 469d799f1ca6496e8bf621130ccdedc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:08.079051 29315 tablet_replica.cc:439] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71c20b836db747028fe52701eb0a75fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } } }
I20260925 14:54:08.079164 29315 tablet_replica.cc:442] T a5943fee3fbe4093aa8576bd3698ddfc P 822ad9e7835a46b69310433699f075d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:54:08.081772 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.079395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.20.199.1:40731)
0925 14:54:08.079455 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.079456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.079464 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:08.079483 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.079484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.079492 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.079552 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.079688 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.080125 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.081191 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.081361 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.081461 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.081596 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.081620 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.081656 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.081715 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.081730 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:08.087059 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.084300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:52733 (local address 127.20.199.1:40731)
0925 14:54:08.084337 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.084339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.084473 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:54:08.084536 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.084536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.084544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.084598 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.084719 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.085208 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.086361 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.086554 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.086618 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.086802 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.086832 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.086880 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.086950 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.086980 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":19}
I20260925 14:54:08.087145 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.084261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.20.199.1:52733)
0925 14:54:08.084413 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.084423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:08.084481 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:08.084605 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:08.084606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:08.084607 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:54:08.084657 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.084659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.085277 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.085278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.086288 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.086291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.086634 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.086635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.086709 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.086723 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.086895 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.086897 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:54:08.086949 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:54:08.087002 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260925 14:54:08.089171 29233 raft_consensus.cc:1261] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Refusing update from remote peer 71c20b836db747028fe52701eb0a75fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.089169 29093 raft_consensus.cc:1261] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Refusing update from remote peer 71c20b836db747028fe52701eb0a75fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.089353 29314 consensus_queue.cc:1035] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.089479 29314 consensus_queue.cc:1035] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.100744 28796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:08.100847 28796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:08.101488 28796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Sending DeleteTablet for 3 replicas of tablet 167843c8971a45b7982e5717e1c9c7f3
I20260925 14:54:08.101732 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet 167843c8971a45b7982e5717e1c9c7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260925 14:54:08.101838 29331 tablet_replica.cc:330] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: stopping tablet replica
I20260925 14:54:08.101899 29331 raft_consensus.cc:2229] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:08.101931 29331 raft_consensus.cc:2258] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.101997 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 167843c8971a45b7982e5717e1c9c7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:59266
I20260925 14:54:08.102103 29330 tablet_replica.cc:330] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: stopping tablet replica
I20260925 14:54:08.102157 29330 raft_consensus.cc:2229] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:08.102149 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet 167843c8971a45b7982e5717e1c9c7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:42418
I20260925 14:54:08.102193 29330 raft_consensus.cc:2258] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.102240 29332 tablet_replica.cc:330] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: stopping tablet replica
I20260925 14:54:08.102268 29331 ts_tablet_manager.cc:1905] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.102291 29332 raft_consensus.cc:2229] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:08.102370 29332 raft_consensus.cc:2258] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.103997 29331 ts_tablet_manager.cc:1918] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:54:08.104049 29331 log.cc:1196] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/167843c8971a45b7982e5717e1c9c7f3
I20260925 14:54:08.104142 29332 ts_tablet_manager.cc:1905] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.104267 29331 ts_tablet_manager.cc:1939] T 167843c8971a45b7982e5717e1c9c7f3 P 822ad9e7835a46b69310433699f075d4: Deleting consensus metadata
I20260925 14:54:08.104666 28783 catalog_manager.cc:4841] TS 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055): tablet 167843c8971a45b7982e5717e1c9c7f3 (table test-table [id=eb5ba981e5764831b6ca2c31d8945500]) successfully deleted
I20260925 14:54:08.105224 29330 ts_tablet_manager.cc:1905] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.105679 29332 ts_tablet_manager.cc:1918] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:54:08.105726 29332 log.cc:1196] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/167843c8971a45b7982e5717e1c9c7f3
I20260925 14:54:08.105917 29332 ts_tablet_manager.cc:1939] T 167843c8971a45b7982e5717e1c9c7f3 P 71c20b836db747028fe52701eb0a75fb: Deleting consensus metadata
I20260925 14:54:08.106950 29330 ts_tablet_manager.cc:1918] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:54:08.107018 29330 log.cc:1196] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/167843c8971a45b7982e5717e1c9c7f3
I20260925 14:54:08.107224 29330 ts_tablet_manager.cc:1939] T 167843c8971a45b7982e5717e1c9c7f3 P 469d799f1ca6496e8bf621130ccdedc1: Deleting consensus metadata
I20260925 14:54:08.107544 28781 catalog_manager.cc:4841] TS 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953): tablet 167843c8971a45b7982e5717e1c9c7f3 (table test-table [id=eb5ba981e5764831b6ca2c31d8945500]) successfully deleted
I20260925 14:54:08.107818 28782 catalog_manager.cc:4841] TS 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731): tablet 167843c8971a45b7982e5717e1c9c7f3 (table test-table [id=eb5ba981e5764831b6ca2c31d8945500]) successfully deleted
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:54:08.130348 28846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:08.124337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.20.199.62:42923)
0925 14:54:08.124394 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.124396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.124480 (+ 84us) server_negotiation.cc:407] Connection header received
0925 14:54:08.124560 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.124561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.124579 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.124629 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:08.125329 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.125904 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.126969 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.127146 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.127192 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.130275 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48316: BlockingRecv error: recv got EOF from 127.0.0.1:48316 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM
I20260925 14:54:08.138296 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.131172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.20.199.62:42923)
0925 14:54:08.131216 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.131218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.131245 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:54:08.131318 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.131319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.131329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.131380 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:08.131808 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.132527 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.133169 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.133484 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.133552 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.135571 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:08.135572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:08.135573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:08.135580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:08.137157 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:08.137527 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:08.137528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:08.137529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:08.137598 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:08.137800 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:08.137800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:08.137801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:08.137929 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:08.138026 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.138129 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.138202 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:08.140044 28796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:08.140223 28796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:08.141589 28933 tablet_service.cc:1467] Processing CreateTablet for tablet c97fc9b0aae84b159776b5c3968762ca (DEFAULT_TABLE table=test-table [id=aa1cb0ed066b4b60a91cafbc0c5f327b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:08.141724 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97fc9b0aae84b159776b5c3968762ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:08.142661 29309 tablet_bootstrap.cc:492] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Bootstrap starting.
I20260925 14:54:08.142742 29073 tablet_service.cc:1467] Processing CreateTablet for tablet c97fc9b0aae84b159776b5c3968762ca (DEFAULT_TABLE table=test-table [id=aa1cb0ed066b4b60a91cafbc0c5f327b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:08.142876 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97fc9b0aae84b159776b5c3968762ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:08.143002 29309 tablet_bootstrap.cc:654] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:08.143563 29309 tablet_bootstrap.cc:492] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: No bootstrap required, opened a new log
I20260925 14:54:08.143620 29309 ts_tablet_manager.cc:1397] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:08.143780 29309 raft_consensus.cc:348] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.143862 29309 raft_consensus.cc:374] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:08.143885 29309 raft_consensus.cc:729] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Initialized, Role: FOLLOWER
I20260925 14:54:08.143915 29309 consensus_queue.cc:260] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.144009 29309 ts_tablet_manager.cc:1428] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:08.144086 29308 tablet_bootstrap.cc:492] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Bootstrap starting.
I20260925 14:54:08.144441 29308 tablet_bootstrap.cc:654] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:08.144454 29213 tablet_service.cc:1467] Processing CreateTablet for tablet c97fc9b0aae84b159776b5c3968762ca (DEFAULT_TABLE table=test-table [id=aa1cb0ed066b4b60a91cafbc0c5f327b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:08.144565 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97fc9b0aae84b159776b5c3968762ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:08.145037 29308 tablet_bootstrap.cc:492] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: No bootstrap required, opened a new log
I20260925 14:54:08.145094 29308 ts_tablet_manager.cc:1397] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:08.145210 29308 raft_consensus.cc:348] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.145272 29308 raft_consensus.cc:374] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:08.145293 29308 raft_consensus.cc:729] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Initialized, Role: FOLLOWER
I20260925 14:54:08.145323 29308 consensus_queue.cc:260] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.145496 29308 ts_tablet_manager.cc:1428] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:08.146252 29307 tablet_bootstrap.cc:492] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Bootstrap starting.
I20260925 14:54:08.146657 29307 tablet_bootstrap.cc:654] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:08.147365 29307 tablet_bootstrap.cc:492] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: No bootstrap required, opened a new log
I20260925 14:54:08.147416 29307 ts_tablet_manager.cc:1397] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:08.147559 29307 raft_consensus.cc:348] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.147627 29307 raft_consensus.cc:374] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:08.147647 29307 raft_consensus.cc:729] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469d799f1ca6496e8bf621130ccdedc1, State: Initialized, Role: FOLLOWER
I20260925 14:54:08.147708 29307 consensus_queue.cc:260] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.147804 29307 ts_tablet_manager.cc:1428] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:08.218045 29315 raft_consensus.cc:482] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:08.218163 29315 raft_consensus.cc:504] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.218355 29315 leader_election.cc:288] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731), 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:08.218571 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97fc9b0aae84b159776b5c3968762ca" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c20b836db747028fe52701eb0a75fb" is_pre_election: true
I20260925 14:54:08.218596 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97fc9b0aae84b159776b5c3968762ca" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1" is_pre_election: true
I20260925 14:54:08.218688 28953 raft_consensus.cc:2454] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 0.
I20260925 14:54:08.218701 29233 raft_consensus.cc:2454] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 0.
I20260925 14:54:08.218854 29026 leader_election.cc:302] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c20b836db747028fe52701eb0a75fb, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:08.218958 29315 raft_consensus.cc:2781] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:08.218998 29315 raft_consensus.cc:482] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:08.219023 29315 raft_consensus.cc:3037] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.219700 29315 raft_consensus.cc:504] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.219844 29315 leader_election.cc:288] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 election: Requested vote from peers 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731), 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953)
I20260925 14:54:08.220017 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97fc9b0aae84b159776b5c3968762ca" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c20b836db747028fe52701eb0a75fb"
I20260925 14:54:08.220011 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97fc9b0aae84b159776b5c3968762ca" candidate_uuid: "822ad9e7835a46b69310433699f075d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469d799f1ca6496e8bf621130ccdedc1"
I20260925 14:54:08.220088 28953 raft_consensus.cc:3037] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.220098 29233 raft_consensus.cc:3037] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.220824 28953 raft_consensus.cc:2454] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 1.
I20260925 14:54:08.220824 29233 raft_consensus.cc:2454] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822ad9e7835a46b69310433699f075d4 in term 1.
I20260925 14:54:08.220995 29025 leader_election.cc:302] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469d799f1ca6496e8bf621130ccdedc1, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:08.221107 29315 raft_consensus.cc:2781] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:08.221177 29315 raft_consensus.cc:686] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 1 LEADER]: Becoming Leader. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Running, Role: LEADER
I20260925 14:54:08.221233 29315 consensus_queue.cc:237] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.221879 28795 catalog_manager.cc:5495] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 822ad9e7835a46b69310433699f075d4 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "822ad9e7835a46b69310433699f075d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:08.237282 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.234988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.20.199.2:46055)
0925 14:54:08.235033 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.235035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.235045 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:08.235093 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.235094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.235105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.235163 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.235300 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.235793 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.236576 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.236795 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.236873 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.237022 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.237057 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.237125 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.237177 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.237200 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:08.238078 28953 raft_consensus.cc:1261] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Refusing update from remote peer 822ad9e7835a46b69310433699f075d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.238077 29233 raft_consensus.cc:1261] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Refusing update from remote peer 822ad9e7835a46b69310433699f075d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.238319 29325 consensus_queue.cc:1035] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.238441 29315 consensus_queue.cc:1035] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.240875 28794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:08.240968 28794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:08.241606 28794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Sending DeleteTablet for 3 replicas of tablet c97fc9b0aae84b159776b5c3968762ca
I20260925 14:54:08.241840 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet c97fc9b0aae84b159776b5c3968762ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:59266
I20260925 14:54:08.241878 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet c97fc9b0aae84b159776b5c3968762ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:42418
I20260925 14:54:08.241928 29330 tablet_replica.cc:330] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: stopping tablet replica
I20260925 14:54:08.241955 29332 tablet_replica.cc:330] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: stopping tablet replica
I20260925 14:54:08.241976 29330 raft_consensus.cc:2229] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:08.241961 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet c97fc9b0aae84b159776b5c3968762ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260925 14:54:08.242000 29332 raft_consensus.cc:2229] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:08.242017 29330 raft_consensus.cc:2258] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.242036 29332 raft_consensus.cc:2258] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.242038 29331 tablet_replica.cc:330] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: stopping tablet replica
I20260925 14:54:08.242093 29331 raft_consensus.cc:2229] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:08.242142 29331 raft_consensus.cc:2258] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.242436 29330 ts_tablet_manager.cc:1905] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.242513 29331 ts_tablet_manager.cc:1905] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.243490 28794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:08.243643 28794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:08.243862 29331 ts_tablet_manager.cc:1918] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:08.243917 29331 log.cc:1196] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/c97fc9b0aae84b159776b5c3968762ca
I20260925 14:54:08.243970 29330 ts_tablet_manager.cc:1918] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:08.244027 29330 log.cc:1196] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/c97fc9b0aae84b159776b5c3968762ca
I20260925 14:54:08.244138 29331 ts_tablet_manager.cc:1939] T c97fc9b0aae84b159776b5c3968762ca P 822ad9e7835a46b69310433699f075d4: Deleting consensus metadata
I20260925 14:54:08.244246 29330 ts_tablet_manager.cc:1939] T c97fc9b0aae84b159776b5c3968762ca P 469d799f1ca6496e8bf621130ccdedc1: Deleting consensus metadata
I20260925 14:54:08.244462 28783 catalog_manager.cc:4841] TS 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055): tablet c97fc9b0aae84b159776b5c3968762ca (table test-table [id=aa1cb0ed066b4b60a91cafbc0c5f327b]) successfully deleted
I20260925 14:54:08.244899 28781 catalog_manager.cc:4841] TS 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953): tablet c97fc9b0aae84b159776b5c3968762ca (table test-table [id=aa1cb0ed066b4b60a91cafbc0c5f327b]) successfully deleted
I20260925 14:54:08.245112 28933 tablet_service.cc:1467] Processing CreateTablet for tablet b9b46ee83ee44b74b2daeeb3065ce2d1 (DEFAULT_TABLE table=test-table [id=1968e9f094fc46f1b9b7230391e1d320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:08.245194 29213 tablet_service.cc:1467] Processing CreateTablet for tablet b9b46ee83ee44b74b2daeeb3065ce2d1 (DEFAULT_TABLE table=test-table [id=1968e9f094fc46f1b9b7230391e1d320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:08.245234 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b46ee83ee44b74b2daeeb3065ce2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:08.245252 29332 ts_tablet_manager.cc:1905] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.245291 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b46ee83ee44b74b2daeeb3065ce2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:08.245569 29073 tablet_service.cc:1467] Processing CreateTablet for tablet b9b46ee83ee44b74b2daeeb3065ce2d1 (DEFAULT_TABLE table=test-table [id=1968e9f094fc46f1b9b7230391e1d320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:08.245654 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b46ee83ee44b74b2daeeb3065ce2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:08.246837 29332 ts_tablet_manager.cc:1918] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:08.246898 29332 log.cc:1196] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/c97fc9b0aae84b159776b5c3968762ca
I20260925 14:54:08.246945 29309 tablet_bootstrap.cc:492] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: Bootstrap starting.
I20260925 14:54:08.247087 29332 ts_tablet_manager.cc:1939] T c97fc9b0aae84b159776b5c3968762ca P 71c20b836db747028fe52701eb0a75fb: Deleting consensus metadata
I20260925 14:54:08.247294 29309 tablet_bootstrap.cc:654] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:08.247524 29308 tablet_bootstrap.cc:492] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Bootstrap starting.
I20260925 14:54:08.247872 29308 tablet_bootstrap.cc:654] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:08.248476 28782 catalog_manager.cc:4841] TS 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731): tablet c97fc9b0aae84b159776b5c3968762ca (table test-table [id=aa1cb0ed066b4b60a91cafbc0c5f327b]) successfully deleted
I20260925 14:54:08.248488 29308 tablet_bootstrap.cc:492] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: No bootstrap required, opened a new log
I20260925 14:54:08.248561 29308 ts_tablet_manager.cc:1397] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:08.248672 29308 raft_consensus.cc:348] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.248731 29308 raft_consensus.cc:374] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:08.248745 29308 raft_consensus.cc:729] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ad9e7835a46b69310433699f075d4, State: Initialized, Role: FOLLOWER
I20260925 14:54:08.248799 29308 consensus_queue.cc:260] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.248899 29308 ts_tablet_manager.cc:1428] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:08.249759 29307 tablet_bootstrap.cc:492] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Bootstrap starting.
I20260925 14:54:08.250171 29307 tablet_bootstrap.cc:654] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:08.250730 29309 tablet_bootstrap.cc:492] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: No bootstrap required, opened a new log
I20260925 14:54:08.250782 29309 ts_tablet_manager.cc:1397] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260925 14:54:08.250875 29309 raft_consensus.cc:348] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.250932 29309 raft_consensus.cc:374] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:08.250952 29309 raft_consensus.cc:729] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c20b836db747028fe52701eb0a75fb, State: Initialized, Role: FOLLOWER
I20260925 14:54:08.250996 29309 consensus_queue.cc:260] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.251073 29309 ts_tablet_manager.cc:1428] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:08.251452 29307 tablet_bootstrap.cc:492] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: No bootstrap required, opened a new log
I20260925 14:54:08.251497 29307 ts_tablet_manager.cc:1397] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:08.251619 29307 raft_consensus.cc:348] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.251678 29307 raft_consensus.cc:374] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:08.251699 29307 raft_consensus.cc:729] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469d799f1ca6496e8bf621130ccdedc1, State: Initialized, Role: FOLLOWER
I20260925 14:54:08.251741 29307 consensus_queue.cc:260] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.251819 29307 ts_tablet_manager.cc:1428] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:08.658860 29313 raft_consensus.cc:482] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:08.658975 29313 raft_consensus.cc:504] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.659327 29313 leader_election.cc:288] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731), 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055)
I20260925 14:54:08.662369 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.659479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.20.199.3:55109)
0925 14:54:08.659608 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.659618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:08.659676 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:08.659861 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:08.659861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:08.659864 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:08.659934 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.659935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.660627 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.660628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.661535 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.661537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.662084 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.662085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.662171 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.662175 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:08.662247 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:54:08.662293 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260925 14:54:08.662390 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.659589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:55109 (local address 127.20.199.1:40731)
0925 14:54:08.659714 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.659717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.659725 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:08.659754 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.659754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.659766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.659857 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:08.660042 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.660543 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.661636 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.662001 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.662103 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.662114 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.662284 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.662325 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260925 14:54:08.662513 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.659550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:46881 (local address 127.20.199.2:46055)
0925 14:54:08.659600 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.659602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.659714 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:54:08.659776 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.659777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.659789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.659836 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:08.659993 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.660481 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.661718 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.662155 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.662237 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.662246 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.662411 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.662458 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:08.662500 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.659499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:46055 (local address 127.20.199.3:46881)
0925 14:54:08.659629 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.659639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:08.659710 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:08.659837 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:08.659838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:08.659841 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:08.659917 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.659919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.660578 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.660579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.661623 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.661626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.662237 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.662238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.662318 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.662322 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:08.662386 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:54:08.662430 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260925 14:54:08.662985 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b46ee83ee44b74b2daeeb3065ce2d1" candidate_uuid: "469d799f1ca6496e8bf621130ccdedc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c20b836db747028fe52701eb0a75fb" is_pre_election: true
I20260925 14:54:08.663054 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b46ee83ee44b74b2daeeb3065ce2d1" candidate_uuid: "469d799f1ca6496e8bf621130ccdedc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ad9e7835a46b69310433699f075d4" is_pre_election: true
I20260925 14:54:08.663122 29093 raft_consensus.cc:2454] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469d799f1ca6496e8bf621130ccdedc1 in term 0.
I20260925 14:54:08.663141 28953 raft_consensus.cc:2454] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469d799f1ca6496e8bf621130ccdedc1 in term 0.
I20260925 14:54:08.663309 29167 leader_election.cc:302] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469d799f1ca6496e8bf621130ccdedc1, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:08.663404 29313 raft_consensus.cc:2781] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:08.663455 29313 raft_consensus.cc:482] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:08.663482 29313 raft_consensus.cc:3037] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.664207 29313 raft_consensus.cc:504] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.664346 29313 leader_election.cc:288] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [CANDIDATE]: Term 1 election: Requested vote from peers 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731), 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055)
I20260925 14:54:08.664484 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b46ee83ee44b74b2daeeb3065ce2d1" candidate_uuid: "469d799f1ca6496e8bf621130ccdedc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ad9e7835a46b69310433699f075d4"
I20260925 14:54:08.664535 29093 raft_consensus.cc:3037] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.665050 29093 raft_consensus.cc:2454] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469d799f1ca6496e8bf621130ccdedc1 in term 1.
I20260925 14:54:08.665154 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b46ee83ee44b74b2daeeb3065ce2d1" candidate_uuid: "469d799f1ca6496e8bf621130ccdedc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c20b836db747028fe52701eb0a75fb"
I20260925 14:54:08.665220 29167 leader_election.cc:302] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469d799f1ca6496e8bf621130ccdedc1, 822ad9e7835a46b69310433699f075d4; no voters:
I20260925 14:54:08.665256 28953 raft_consensus.cc:3037] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:08.665323 29313 raft_consensus.cc:2781] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:08.665467 29313 raft_consensus.cc:686] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 LEADER]: Becoming Leader. State: Replica: 469d799f1ca6496e8bf621130ccdedc1, State: Running, Role: LEADER
I20260925 14:54:08.665535 29313 consensus_queue.cc:237] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } }
I20260925 14:54:08.666098 28953 raft_consensus.cc:2454] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469d799f1ca6496e8bf621130ccdedc1 in term 1.
I20260925 14:54:08.666337 28794 catalog_manager.cc:5495] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3). New cstate: current_term: 1 leader_uuid: "469d799f1ca6496e8bf621130ccdedc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469d799f1ca6496e8bf621130ccdedc1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40953 } health_report { overall_health: HEALTHY } } }
I20260925 14:54:08.691452 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.688735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.20.199.3:40953)
0925 14:54:08.688894 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.688895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.688902 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:54:08.688939 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.688940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.688964 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.689050 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.689171 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.689747 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.690508 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.690724 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.690797 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.690995 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.691092 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.691173 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.691334 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.691358 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260925 14:54:08.695897 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.693763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:43715 (local address 127.20.199.1:40731)
0925 14:54:08.693815 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.693817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.693895 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:54:08.693936 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.693936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.693945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.693993 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.694106 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.694533 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.695261 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.695521 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.695584 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.695714 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.695747 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.695793 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.695827 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.695845 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:54:08.695926 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.693690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.20.199.3:43715)
0925 14:54:08.693827 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.693837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:08.693895 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:08.694006 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:08.694007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:08.694008 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:54:08.694065 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.694066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.694599 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.694599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.695184 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.695185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.695591 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.695592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.695630 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.695642 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.695794 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.695795 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:54:08.695826 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:54:08.695857 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260925 14:54:08.698215 29093 raft_consensus.cc:1261] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Refusing update from remote peer 469d799f1ca6496e8bf621130ccdedc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.698266 28953 raft_consensus.cc:1261] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Refusing update from remote peer 469d799f1ca6496e8bf621130ccdedc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:08.698477 29386 consensus_queue.cc:1035] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c20b836db747028fe52701eb0a75fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.698597 29313 consensus_queue.cc:1035] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822ad9e7835a46b69310433699f075d4" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:08.705751 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.703313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:52509 (local address 127.20.199.3:40953)
0925 14:54:08.703357 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:08.703359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:08.703477 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:54:08.703525 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:08.703525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:08.703537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:08.703607 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:08.703727 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.704180 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.704989 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.705285 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.705357 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.705520 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.705554 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.705606 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:08.705660 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:54:08.705683 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:54:08.705785 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.703251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:40953 (local address 127.20.199.1:52509)
0925 14:54:08.703406 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.703417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:08.703473 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:08.703621 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:08.703622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:08.703624 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:54:08.703676 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.703677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.704269 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.704270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.704898 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:08.704900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:08.705356 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:08.705357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:08.705442 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:08.705448 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:54:08.705608 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:54:08.705610 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:54:08.705656 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:54:08.705710 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":108,"threads_started":1}
I20260925 14:54:08.712200 28794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:08.712320 28794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:08.712896 28794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdc0ee11469461fb325f5442f4e94cb: Sending DeleteTablet for 3 replicas of tablet b9b46ee83ee44b74b2daeeb3065ce2d1
I20260925 14:54:08.713222 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b46ee83ee44b74b2daeeb3065ce2d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260925 14:54:08.713231 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b46ee83ee44b74b2daeeb3065ce2d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:42418
I20260925 14:54:08.713331 29331 tablet_replica.cc:330] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: stopping tablet replica
I20260925 14:54:08.713336 29332 tablet_replica.cc:330] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: stopping tablet replica
I20260925 14:54:08.713379 29332 raft_consensus.cc:2229] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:08.713392 29331 raft_consensus.cc:2229] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:08.713443 29332 raft_consensus.cc:2258] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.713449 29331 raft_consensus.cc:2258] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.713740 29332 ts_tablet_manager.cc:1905] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 71c20b836db747028fe52701eb0a75fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.713861 29331 ts_tablet_manager.cc:1905] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.713990 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 28868
I20260925 14:54:08.715519 29331 ts_tablet_manager.cc:1918] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:54:08.715584 29331 log.cc:1196] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/b9b46ee83ee44b74b2daeeb3065ce2d1
I20260925 14:54:08.715821 29331 ts_tablet_manager.cc:1939] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 822ad9e7835a46b69310433699f075d4: Deleting consensus metadata
I20260925 14:54:08.716130 28783 catalog_manager.cc:4841] TS 822ad9e7835a46b69310433699f075d4 (127.20.199.2:46055): tablet b9b46ee83ee44b74b2daeeb3065ce2d1 (table test-table [id=1968e9f094fc46f1b9b7230391e1d320]) successfully deleted
I20260925 14:54:08.721700 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b46ee83ee44b74b2daeeb3065ce2d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:08 UTC) from {username='oryx', principal='oryx/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:59266
I20260925 14:54:08.721812 29330 tablet_replica.cc:330] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: stopping tablet replica
I20260925 14:54:08.721872 29330 raft_consensus.cc:2229] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:08.721972 29330 raft_consensus.cc:2258] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:08.722375 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29008
W20260925 14:54:08.722375 28782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:40731 (error 108)
I20260925 14:54:08.722407 29330 ts_tablet_manager.cc:1905] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:08.723311 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.723133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:40731 (local address 127.0.0.1:42444)
0925 14:54:08.723227 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.723263 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":35,"threads_started":1}
W20260925 14:54:08.723387 28782 catalog_manager.cc:4584] TS 71c20b836db747028fe52701eb0a75fb (127.20.199.1:40731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9b46ee83ee44b74b2daeeb3065ce2d1: Network error: Client connection negotiation failed: client connection to 127.20.199.1:40731: connect: Connection refused (error 111)
I20260925 14:54:08.723788 29330 ts_tablet_manager.cc:1918] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:54:08.723837 29330 log.cc:1196] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/b9b46ee83ee44b74b2daeeb3065ce2d1
I20260925 14:54:08.724037 29330 ts_tablet_manager.cc:1939] T b9b46ee83ee44b74b2daeeb3065ce2d1 P 469d799f1ca6496e8bf621130ccdedc1: Deleting consensus metadata
I20260925 14:54:08.724354 28781 catalog_manager.cc:4841] TS 469d799f1ca6496e8bf621130ccdedc1 (127.20.199.3:40953): tablet b9b46ee83ee44b74b2daeeb3065ce2d1 (table test-table [id=1968e9f094fc46f1b9b7230391e1d320]) successfully deleted
I20260925 14:54:08.729046 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29148
I20260925 14:54:08.734331 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 28764
2026-09-25T14:54:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2750 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[29405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[29405](info): set up 2 sockets
Sep 25 14:54:08 dist-test-slave-8qt6 krb5kdc[29405](info): commencing operation
krb5kdc: starting...
W20260925 14:54:09.766091 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:54:09 dist-test-slave-8qt6 krb5kdc[29405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:54:09.880470 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38675
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:09.962643 29418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:09.962953 29418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:09.963003 29418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:09.964181 29418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:09.964247 29418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:09.964272 29418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:09.964304 29418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:09.964331 29418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:09.965883 29418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:09.966162 29418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:09.966377 29418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:09.971176 29426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:09.971192 29423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:09.971246 29424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:09.971544 29418 server_base.cc:1034] running on GCE node
I20260925 14:54:09.971817 29418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:09.972093 29418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:09.973268 29418 hybrid_clock.cc:648] HybridClock initialized: now 1790348049973248 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:54:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1243 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:10 dist-test-slave-8qt6 krb5kdc[29431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:10 dist-test-slave-8qt6 krb5kdc[29431](info): set up 2 sockets
Sep 25 14:54:10 dist-test-slave-8qt6 krb5kdc[29431](info): commencing operation
krb5kdc: starting...
W20260925 14:54:11.030228 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:54:11.146143 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38677
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:11.225550 29444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:11.225771 29444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:11.225814 29444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:11.226949 29444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:11.227015 29444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:11.227041 29444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:11.227072 29444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:11.227108 29444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:11.228557 29444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:11.228822 29444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:11.229044 29444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:11.233719 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.233716 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.233731 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:11.233952 29444 server_base.cc:1034] running on GCE node
I20260925 14:54:11.234165 29444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:11.234404 29444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:11.235539 29444 hybrid_clock.cc:648] HybridClock initialized: now 1790348051235527 us; error 28 us; skew 500 ppm
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:11.243871 29444 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:54:11.244319 29444 webserver.cc:460] Webserver started at http://127.20.199.62:38141/ using document root <none> and password file <none>
I20260925 14:54:11.244516 29444 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:11.244555 29444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:11.244642 29444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:11.245697 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "6353d8a951c44af19c544f6686efc7cc"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ef115cdebe2bb04d814649b3ff6d82ed"
server_key_iv: "694c5b036ab7ed29be869771d823dcd2"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.245994 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "6353d8a951c44af19c544f6686efc7cc"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ef115cdebe2bb04d814649b3ff6d82ed"
server_key_iv: "694c5b036ab7ed29be869771d823dcd2"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.247246 29444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:11.248024 29459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.248229 29444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:11.248333 29444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "6353d8a951c44af19c544f6686efc7cc"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ef115cdebe2bb04d814649b3ff6d82ed"
server_key_iv: "694c5b036ab7ed29be869771d823dcd2"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.248407 29444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:11.257377 29444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:11.259168 29444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:11.259320 29444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:11.263093 29444 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:38677
I20260925 14:54:11.263123 29511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:38677 every 8 connection(s)
I20260925 14:54:11.263453 29444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:11.264231 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:11.266482 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Bootstrap starting.
I20260925 14:54:11.267035 29512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:11.267313 29512 log.cc:824] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:11.267963 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: No bootstrap required, opened a new log
I20260925 14:54:11.269544 29512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.269670 29512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:11.269691 29512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6353d8a951c44af19c544f6686efc7cc, State: Initialized, Role: FOLLOWER
I20260925 14:54:11.269877 29512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.269954 29512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:11.269994 29512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:11.270041 29512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:11.270514 29512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.270620 29512 leader_election.cc:302] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6353d8a951c44af19c544f6686efc7cc; no voters:
I20260925 14:54:11.270816 29512 leader_election.cc:288] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:11.270835 29515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:11.270953 29515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 LEADER]: Becoming Leader. State: Replica: 6353d8a951c44af19c544f6686efc7cc, State: Running, Role: LEADER
I20260925 14:54:11.270946 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29444
I20260925 14:54:11.271057 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:54:11.271072 29512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:11.271111 29515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.271248 21276 external_mini_cluster.cc:1428] Setting key c53b76f494019a67ab6c6399d547a8c7
I20260925 14:54:11.271425 29516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6353d8a951c44af19c544f6686efc7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } } }
I20260925 14:54:11.271502 29516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:11.271565 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6353d8a951c44af19c544f6686efc7cc. Latest consensus state: current_term: 1 leader_uuid: "6353d8a951c44af19c544f6686efc7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } } }
I20260925 14:54:11.271651 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:11.271713 29522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:11.272380 29522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:11.274031 29522 catalog_manager.cc:1347] Generated new cluster ID: 664f0ab1cb1b49009462ab37023b32fa
I20260925 14:54:11.274091 29522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:11.281436 29522 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:11.282084 29522 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:11.282810 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.271819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.20.199.62:38677)
0925 14:54:11.272007 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:11.272016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:11.272053 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:54:11.272168 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:11.272174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:11.272230 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:11.272983 (+ 753us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:11.273202 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.274753 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.275858 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.276481 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.276607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.279272 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:11.279276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:11.279276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:11.279288 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:11.281195 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.281597 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.281600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.281601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.281698 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.281921 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.281922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.281922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.282187 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:11.282328 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:11.282463 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 14:54:11.282600 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260925 14:54:11.285537 29522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Generated new TSK 0
I20260925 14:54:11.285691 29522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:54:11.316218 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38677
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:11.395530 29538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:11.395759 29538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:11.395799 29538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:11.396973 29538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:11.397040 29538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:11.397085 29538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:11.398617 29538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:11.398892 29538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:11.399132 29538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:11.404044 29546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.404037 29544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.404037 29543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:11.404124 29538 server_base.cc:1034] running on GCE node
I20260925 14:54:11.404486 29538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:11.404739 29538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:11.405876 29538 hybrid_clock.cc:648] HybridClock initialized: now 1790348051405865 us; error 27 us; skew 500 ppm
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:11.408209 29538 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:54:11.408649 29538 webserver.cc:460] Webserver started at http://127.20.199.1:40079/ using document root <none> and password file <none>
I20260925 14:54:11.408879 29538 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:11.408957 29538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:11.409109 29538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:11.410211 29538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "07f9c36952044d81b7b35df3d469d987"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ca23f5d4f138d98707b62b41391f939b"
server_key_iv: "2274886b81a91ac99027f7b7c7670d28"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.410562 29538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "07f9c36952044d81b7b35df3d469d987"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ca23f5d4f138d98707b62b41391f939b"
server_key_iv: "2274886b81a91ac99027f7b7c7670d28"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.411875 29538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:11.412621 29553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.412807 29538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:11.412910 29538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "07f9c36952044d81b7b35df3d469d987"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ca23f5d4f138d98707b62b41391f939b"
server_key_iv: "2274886b81a91ac99027f7b7c7670d28"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.412992 29538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:11.443291 29538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:11.445494 29538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:11.445667 29538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:11.445909 29538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:11.446233 29538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:11.446287 29538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.446328 29538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:11.446372 29538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.452277 29538 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:36729
I20260925 14:54:11.452342 29666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:36729 every 8 connection(s)
I20260925 14:54:11.452651 29538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:11.461149 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29538
I20260925 14:54:11.461256 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:11.461374 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.453367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38221 (local address 127.20.199.62:38677)
0925 14:54:11.453460 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:11.453463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:11.453670 (+ 207us) server_negotiation.cc:407] Connection header received
0925 14:54:11.454164 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:11.454165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:11.454174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:11.454228 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:11.455180 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.455672 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.456625 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.456813 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.456862 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.459004 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:11.459005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:11.459005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:11.459011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:11.460196 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.460543 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.460544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.460544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.460591 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.460798 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.460799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.460799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.460894 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:11.460967 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:11.461183 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:54:11.461277 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:54:11.461512 21276 external_mini_cluster.cc:1428] Setting key e009dffedb12f3ad2d9c016b1335b9b1
I20260925 14:54:11.461594 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.453351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:38221)
0925 14:54:11.453547 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:11.453584 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:11.454082 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:11.454285 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:11.454288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:11.454887 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:11.455102 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:11.455108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.455753 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.455754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:11.456552 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:11.456554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.456886 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.456887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:11.456991 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.457563 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:11.457571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:11.458927 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:11.460268 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:11.460276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:11.460277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:11.460479 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:11.460648 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:11.460648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:11.460648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:11.460740 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:11.460952 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:11.460956 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:11.461061 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 14:54:11.461139 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:54:11.461300 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
I20260925 14:54:11.462175 29667 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38677
I20260925 14:54:11.462291 29667 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:11.462474 29667 heartbeater.cc:511] Master 127.20.199.62:38677 requested a full tablet report, sending...
I20260925 14:54:11.463028 29476 ts_manager.cc:194] Registered new tserver with Master: 07f9c36952044d81b7b35df3d469d987 (127.20.199.1:36729)
I20260925 14:54:11.463927 29476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:38221
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:54:11.488116 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38677
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:11.566879 29672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:11.567127 29672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:11.567165 29672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:11.568311 29672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:11.568378 29672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:11.568420 29672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:11.569857 29672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:11.570114 29672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:11.570339 29672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:11.575623 29677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.575621 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.575618 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:11.575838 29672 server_base.cc:1034] running on GCE node
I20260925 14:54:11.576074 29672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:11.576318 29672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:11.577487 29672 hybrid_clock.cc:648] HybridClock initialized: now 1790348051577465 us; error 36 us; skew 500 ppm
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:11.579823 29672 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:54:11.580251 29672 webserver.cc:460] Webserver started at http://127.20.199.2:35889/ using document root <none> and password file <none>
I20260925 14:54:11.580487 29672 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:11.580531 29672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:11.580672 29672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:11.581784 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "4440fba4b75241afa5f0875e7ef73ce0"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "4fbebfa480891b679c92a80fcb24117c"
server_key_iv: "b28de8f17e01bcc67d8a23e2bc635f3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.582134 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "4440fba4b75241afa5f0875e7ef73ce0"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "4fbebfa480891b679c92a80fcb24117c"
server_key_iv: "b28de8f17e01bcc67d8a23e2bc635f3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.583503 29672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:11.584268 29687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.584466 29672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:11.584559 29672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "4440fba4b75241afa5f0875e7ef73ce0"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "4fbebfa480891b679c92a80fcb24117c"
server_key_iv: "b28de8f17e01bcc67d8a23e2bc635f3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.584612 29672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:11.592526 29672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:11.594381 29672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:11.594496 29672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:11.594681 29672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:11.594996 29672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:11.595029 29672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.595050 29672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:11.595098 29672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.600965 29672 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:43669
I20260925 14:54:11.601006 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:43669 every 8 connection(s)
I20260925 14:54:11.601473 29672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:11.602763 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29672
I20260925 14:54:11.602880 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:11.603062 21276 external_mini_cluster.cc:1428] Setting key 6594958eaaa3314db6b88225e10e3b56
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:11.611495 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.602220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:45175 (local address 127.20.199.62:38677)
0925 14:54:11.602289 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:11.602291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:11.602497 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:54:11.603169 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:11.603170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:11.603181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:11.603244 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:11.604248 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.604728 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.605797 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.606102 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.606172 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.608530 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:11.608532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:11.608533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:11.608541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:11.610205 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.610587 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.610589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.610589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.610644 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.610873 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.610873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.610874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.611017 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:11.611096 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:11.611322 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 14:54:11.611404 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:54:11.611562 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.602181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:45175)
0925 14:54:11.602378 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:11.602414 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:11.603063 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:11.603318 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:11.603321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:11.603959 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:11.604146 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:11.604150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.604828 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.604829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:11.605691 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:11.605694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.606178 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.606179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:11.606285 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.606806 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:11.606814 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:11.608437 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:11.610303 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:11.610309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:11.610310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:11.610514 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:11.610716 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:11.610717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:11.610717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:11.610805 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:11.611087 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:11.611091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:11.611195 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 14:54:11.611273 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:54:11.611382 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:54:11.612036 29801 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38677
I20260925 14:54:11.612136 29801 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:11.612304 29801 heartbeater.cc:511] Master 127.20.199.62:38677 requested a full tablet report, sending...
I20260925 14:54:11.612689 29476 ts_manager.cc:194] Registered new tserver with Master: 4440fba4b75241afa5f0875e7ef73ce0 (127.20.199.2:43669)
I20260925 14:54:11.613072 29476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:45175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:54:11.628062 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38677
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:11.707952 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:11.708148 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:11.708170 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:11.709246 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:11.709282 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:11.709307 29806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:11.710703 29806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:11.710882 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:11.711046 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:11.715988 29814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.716019 29811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.716053 29812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:11.716099 29806 server_base.cc:1034] running on GCE node
I20260925 14:54:11.716428 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:11.716715 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:11.717877 29806 hybrid_clock.cc:648] HybridClock initialized: now 1790348051717862 us; error 49 us; skew 500 ppm
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:11.720323 29806 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:54:11.720737 29806 webserver.cc:460] Webserver started at http://127.20.199.3:45713/ using document root <none> and password file <none>
I20260925 14:54:11.720957 29806 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:11.721024 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:11.721172 29806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:11.722270 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "b81d58d808b64a19889c8d5bee5fe9c7"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "365beca09c06b8bf224374fe47e0bf41"
server_key_iv: "1274844cdaa8af454cf9cff5cb808769"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.722640 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "b81d58d808b64a19889c8d5bee5fe9c7"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "365beca09c06b8bf224374fe47e0bf41"
server_key_iv: "1274844cdaa8af454cf9cff5cb808769"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.724013 29806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:11.724835 29821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.725047 29806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:11.725160 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "b81d58d808b64a19889c8d5bee5fe9c7"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "365beca09c06b8bf224374fe47e0bf41"
server_key_iv: "1274844cdaa8af454cf9cff5cb808769"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.725265 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:11.739943 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:11.742344 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:11.742514 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:11.742761 29806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:11.743090 29806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:11.743144 29806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.743184 29806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:11.743212 29806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.748903 29806 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:46043
I20260925 14:54:11.748976 29934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:46043 every 8 connection(s)
I20260925 14:54:11.749300 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:11.752686 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29806
I20260925 14:54:11.752774 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:11.753017 21276 external_mini_cluster.cc:1428] Setting key 1c71c68ab62c929508695ed46dca956b
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:11.758617 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.750084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:53825 (local address 127.20.199.62:38677)
0925 14:54:11.750155 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:11.750157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:11.750372 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:54:11.750888 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:11.750890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:11.750902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:11.750952 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:11.751922 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.752507 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.754006 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.754214 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.754272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.756314 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:11.756315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:11.756316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:11.756322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:11.757536 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.757879 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.757880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.757880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.757936 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:11.758146 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:11.758147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:11.758147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:11.758241 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:11.758289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:11.758501 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 14:54:11.758557 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:11.758824 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.750081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:53825)
0925 14:54:11.750244 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:11.750285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:11.750767 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:11.751021 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:11.751030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:11.751619 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:11.751808 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:11.751812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.752602 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.752603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:11.753912 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:11.753916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:11.754284 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:11.754285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:11.754383 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:11.754919 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:11.754927 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:11.756228 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:11.757602 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:11.757608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:11.757609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:11.757805 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:11.758003 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:11.758004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:11.758004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:11.758093 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:11.758292 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:11.758296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:11.758399 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 14:54:11.758476 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:54:11.758582 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260925 14:54:11.759348 29935 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38677
I20260925 14:54:11.759446 29935 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:11.759611 29935 heartbeater.cc:511] Master 127.20.199.62:38677 requested a full tablet report, sending...
I20260925 14:54:11.760015 29476 ts_manager.cc:194] Registered new tserver with Master: b81d58d808b64a19889c8d5bee5fe9c7 (127.20.199.3:46043)
I20260925 14:54:11.760337 29476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:53825
I20260925 14:54:11.764765 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab.
Entry for principal oryx/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab.
WARNING: no policy specified for oryx/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab.
Entry for principal oryx/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab.
WARNING: no policy specified for oryx/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab.
Entry for principal oryx/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab.
I20260925 14:54:11.836090 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29538
I20260925 14:54:11.840453 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29672
I20260925 14:54:11.844623 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29806
I20260925 14:54:11.848696 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29444
I20260925 14:54:11.853514 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38677
--webserver_interface=127.20.199.62
--webserver_port=38141
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:11.934453 29944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:11.934806 29944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:11.934841 29944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:11.936933 29944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:11.937007 29944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:11.937032 29944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:11.937060 29944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:11.937091 29944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:11.939561 29944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=38141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:11.939872 29944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:11.940169 29944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:11.946518 29952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.946763 29949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:11.946578 29950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:11.946646 29944 server_base.cc:1034] running on GCE node
I20260925 14:54:11.947525 29944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:11.947773 29944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:11.948938 29944 hybrid_clock.cc:648] HybridClock initialized: now 1790348051948926 us; error 33 us; skew 500 ppm
Sep 25 14:54:11 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:11.951469 29944 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:54:11.951912 29944 webserver.cc:460] Webserver started at http://127.20.199.62:38141/ using document root <none> and password file <none>
I20260925 14:54:11.952144 29944 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:11.952198 29944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:11.953644 29944 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:11.954344 29959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:11.954571 29944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:11.954643 29944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "6353d8a951c44af19c544f6686efc7cc"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ef115cdebe2bb04d814649b3ff6d82ed"
server_key_iv: "694c5b036ab7ed29be869771d823dcd2"
server_key_version: "encryptionkey@0"
I20260925 14:54:11.955013 29944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:11.967929 29944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:11.969995 29944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:11.970134 29944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:11.974160 29944 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:38677
I20260925 14:54:11.974241 30011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:38677 every 8 connection(s)
I20260925 14:54:11.974591 29944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:11.975996 30012 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:54:11.976585 30012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Bootstrap starting.
I20260925 14:54:11.978472 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 29944
I20260925 14:54:11.978845 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:36729
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:38677
--webserver_port=40079
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:11.985082 30012 log.cc:824] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:11.986805 30012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:54:11.987087 30012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Bootstrap complete.
I20260925 14:54:11.989282 30012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.989593 30012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6353d8a951c44af19c544f6686efc7cc, State: Initialized, Role: FOLLOWER
I20260925 14:54:11.989799 30012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.989889 30012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:11.989943 30012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:11.989990 30012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:54:11.990940 30012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.991056 30012 leader_election.cc:302] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6353d8a951c44af19c544f6686efc7cc; no voters:
I20260925 14:54:11.991242 30012 leader_election.cc:288] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:54:11.991307 30016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:54:11.991492 30012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:11.991529 30016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [term 2 LEADER]: Becoming Leader. State: Replica: 6353d8a951c44af19c544f6686efc7cc, State: Running, Role: LEADER
I20260925 14:54:11.991760 30016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } }
I20260925 14:54:11.992031 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6353d8a951c44af19c544f6686efc7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } } }
I20260925 14:54:11.992094 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:11.992324 30018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6353d8a951c44af19c544f6686efc7cc. Latest consensus state: current_term: 2 leader_uuid: "6353d8a951c44af19c544f6686efc7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6353d8a951c44af19c544f6686efc7cc" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38677 } } }
I20260925 14:54:11.992384 30018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:11.992519 30023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:11.993086 30023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:11.993274 30023 catalog_manager.cc:1259] Loaded cluster ID: 664f0ab1cb1b49009462ab37023b32fa
I20260925 14:54:11.993453 30023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:11.994222 30023 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:11.994331 30023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6353d8a951c44af19c544f6686efc7cc: Loaded TSK: 0
I20260925 14:54:11.994459 30023 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:54:12.071389 30014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:12.071609 30014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:12.071643 30014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:12.072809 30014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:12.072892 30014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:12.072938 30014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:12.074477 30014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.1.keytab
--principal=oryx/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:36729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=40079
--tserver_master_addrs=127.20.199.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:12.074725 30014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:12.074945 30014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:12.080744 30040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:12.080734 30039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:12.081087 30042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:12.081202 30014 server_base.cc:1034] running on GCE node
I20260925 14:54:12.081477 30014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:12.081810 30014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:12.083024 30014 hybrid_clock.cc:648] HybridClock initialized: now 1790348052083002 us; error 43 us; skew 500 ppm
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348052, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:12.085861 30014 init.cc:377] Logged in from keytab as oryx/127.20.199.1@KRBTEST.COM (short username oryx)
I20260925 14:54:12.086393 30014 webserver.cc:460] Webserver started at http://127.20.199.1:40079/ using document root <none> and password file <none>
I20260925 14:54:12.086678 30014 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:12.086767 30014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:12.088552 30014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:12.089301 30049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.089563 30014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:12.089659 30014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "07f9c36952044d81b7b35df3d469d987"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "ca23f5d4f138d98707b62b41391f939b"
server_key_iv: "2274886b81a91ac99027f7b7c7670d28"
server_key_version: "encryptionkey@0"
I20260925 14:54:12.090109 30014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:12.107589 30014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:12.110280 30014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:12.110461 30014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:12.110776 30014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:12.111153 30014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:12.111222 30014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.111258 30014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:12.111276 30014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.118685 30014 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:36729
I20260925 14:54:12.118754 30162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:36729 every 8 connection(s)
I20260925 14:54:12.119174 30014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:12.124265 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30014
I20260925 14:54:12.124581 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:43669
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:38677
--webserver_port=35889
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.128756 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.119950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:42221)
0925 14:54:12.120135 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.120185 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.120711 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.121180 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.121183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.121827 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.122014 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.122019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.123135 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.123136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.123942 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.123944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.124917 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.124918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.125062 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.125712 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.125723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.128524 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
W20260925 14:54:12.128847 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.120000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:42221 (local address 127.20.199.62:38677)
0925 14:54:12.120237 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.120245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.120283 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:54:12.120945 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.120951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.121005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.121141 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.122135 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.123031 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.124043 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.124720 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.124826 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.128662 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:42221: BlockingRecv error: recv got EOF from 127.20.199.1:42221 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
W20260925 14:54:12.129508 30163 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:38677 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.199.62:38677: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.135639 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.129754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:58465)
0925 14:54:12.129884 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.129896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.129945 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.130102 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.130104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.130274 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.130330 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.130331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.131233 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.131234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.132155 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.132158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.132661 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.132663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.132739 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.133228 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.133233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.135548 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 14:54:12.135753 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.129840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58465 (local address 127.20.199.62:38677)
0925 14:54:12.129893 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.129895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.129931 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:54:12.130001 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.130002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.130015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.130081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.130410 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.131125 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.132249 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.132573 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.132650 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.135671 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:58465: BlockingRecv error: recv got EOF from 127.20.199.1:58465 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.141536 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.136019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:48571)
0925 14:54:12.136192 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.136246 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.136309 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.136459 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.136460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.136625 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.136684 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.136685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.137598 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.137599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.138516 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.138518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.138977 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.138978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.139038 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.139530 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.139544 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.141458 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
W20260925 14:54:12.141610 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.136183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:48571 (local address 127.20.199.62:38677)
0925 14:54:12.136221 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.136223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.136317 (+ 94us) server_negotiation.cc:407] Connection header received
0925 14:54:12.136366 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.136366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.136378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.136442 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.136763 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.137521 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.138593 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.138882 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.138947 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.141542 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:48571: BlockingRecv error: recv got EOF from 127.20.199.1:48571 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:54:12.142033 30163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:54:12.209574 30167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:12.209810 30167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:12.209848 30167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:12.211014 30167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:12.211081 30167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:12.211126 30167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:12.212561 30167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.2.keytab
--principal=oryx/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:43669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=35889
--tserver_master_addrs=127.20.199.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:12.212841 30167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:12.213083 30167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:12.217944 30173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:12.217991 30174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:12.218029 30167 server_base.cc:1034] running on GCE node
W20260925 14:54:12.217991 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:12.218341 30167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:12.218531 30167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:12.219726 30167 hybrid_clock.cc:648] HybridClock initialized: now 1790348052219712 us; error 26 us; skew 500 ppm
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348052, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:12.222353 30167 init.cc:377] Logged in from keytab as oryx/127.20.199.2@KRBTEST.COM (short username oryx)
I20260925 14:54:12.222748 30167 webserver.cc:460] Webserver started at http://127.20.199.2:35889/ using document root <none> and password file <none>
I20260925 14:54:12.222941 30167 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:12.222991 30167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:12.224229 30167 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:12.224809 30183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.225119 30167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:12.225224 30167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "4440fba4b75241afa5f0875e7ef73ce0"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "4fbebfa480891b679c92a80fcb24117c"
server_key_iv: "b28de8f17e01bcc67d8a23e2bc635f3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:12.225610 30167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:12.250195 30167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:12.252130 30167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:12.252292 30167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:12.252527 30167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:12.252838 30167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:12.252907 30167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.252949 30167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:12.252976 30167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.259017 30167 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:43669
I20260925 14:54:12.259060 30296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:43669 every 8 connection(s)
I20260925 14:54:12.259384 30167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:12.260295 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30167
I20260925 14:54:12.260591 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:46043
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:38677
--webserver_port=45713
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:54:12.267912 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.260048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57797 (local address 127.20.199.62:38677)
0925 14:54:12.260121 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.260123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.260358 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:54:12.260982 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.260983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.261001 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.261080 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.262207 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.262665 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.264249 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.264474 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.264528 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.267846 (+ 3318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:57797: BlockingRecv error: recv got EOF from 127.20.199.2:57797 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:12.267961 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.260055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:57797)
0925 14:54:12.260225 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.260265 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.260880 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.261131 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.261134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.261821 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.262103 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.262110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.262780 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.262781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.264156 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.264160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.264557 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.264558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.264718 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.265191 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.265201 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.267759 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":110,"threads_started":1}
W20260925 14:54:12.268293 30297 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:38677 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.199.62:38677: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.272710 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.268527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:42159)
0925 14:54:12.268558 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.268574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.268629 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.268777 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.268778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.268901 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.268941 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.268942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.269627 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.269628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.270123 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.270124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.270539 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.270540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.270609 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.271047 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.271052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.272612 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 14:54:12.272773 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.268579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:42159 (local address 127.20.199.62:38677)
0925 14:54:12.268643 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.268646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.268654 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:12.268683 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.268684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.268701 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.268774 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.269025 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.269551 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.270225 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.270453 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.270539 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.272709 (+ 2170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:42159: BlockingRecv error: recv got EOF from 127.20.199.2:42159 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.277453 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.273203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:54151)
0925 14:54:12.273250 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.273262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.273320 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.273469 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.273470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.273590 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.273612 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.273613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.274246 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.274247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.274815 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.274817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.275202 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.275203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.275255 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.275672 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.275677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.277356 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:54:12.277556 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.273251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:54151 (local address 127.20.199.62:38677)
0925 14:54:12.273303 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.273305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.273314 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:12.273377 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.273378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.273387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.273465 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.273704 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.274143 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.274915 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.275121 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.275198 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.277484 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:54151: BlockingRecv error: recv got EOF from 127.20.199.2:54151 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 14:54:12.277695 30297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:54:12.345276 30300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:12.345546 30300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:12.345584 30300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:12.346726 30300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:12.346791 30300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:12.346835 30300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:12.348227 30300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/krb5kdc/oryx_127.20.199.3.keytab
--principal=oryx/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:46043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=45713
--tserver_master_addrs=127.20.199.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:12.348479 30300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:12.348712 30300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:12.353722 30306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:12.353731 30305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:12.353762 30300 server_base.cc:1034] running on GCE node
W20260925 14:54:12.353772 30308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:12.354180 30300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:12.354434 30300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:12.355571 30300 hybrid_clock.cc:648] HybridClock initialized: now 1790348052355553 us; error 35 us; skew 500 ppm
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348052, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:12.358019 30300 init.cc:377] Logged in from keytab as oryx/127.20.199.3@KRBTEST.COM (short username oryx)
I20260925 14:54:12.358505 30300 webserver.cc:460] Webserver started at http://127.20.199.3:45713/ using document root <none> and password file <none>
I20260925 14:54:12.358690 30300 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:12.358739 30300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:12.360001 30300 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:12.360677 30315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.360904 30300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:12.361037 30300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "b81d58d808b64a19889c8d5bee5fe9c7"
format_stamp: "Formatted at 2026-09-25 14:54:11 on dist-test-slave-8qt6"
server_key: "365beca09c06b8bf224374fe47e0bf41"
server_key_iv: "1274844cdaa8af454cf9cff5cb808769"
server_key_version: "encryptionkey@0"
I20260925 14:54:12.361560 30300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:12.370268 30300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:12.372586 30300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:12.372720 30300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:12.372905 30300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:12.373154 30300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:12.373188 30300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.373214 30300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:12.373227 30300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:12.379462 30300 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:46043
I20260925 14:54:12.379515 30428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:46043 every 8 connection(s)
I20260925 14:54:12.379866 30300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.385841 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30300
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:54:12.387296 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.380620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:41255 (local address 127.20.199.62:38677)
0925 14:54:12.380697 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.380701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.380936 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:54:12.381494 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.381495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.381508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.381575 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.382517 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.382999 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.383951 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.384131 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.384228 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.387238 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:41255: BlockingRecv error: recv got EOF from 127.20.199.3:41255 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:54:12.387389 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.380610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:41255)
0925 14:54:12.380809 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.380845 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.381332 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.381649 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.381652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.382226 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.382418 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.382423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.383096 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.383097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.383863 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.383864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.384235 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.384236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.384359 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.384861 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.384869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.387178 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260925 14:54:12.387583 30429 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:38677 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.199.62:38677: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.391788 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.387791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:59823)
0925 14:54:12.387838 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.387848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.387885 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.387991 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.387991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.388115 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.388157 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.388158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.388755 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.388756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.389238 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.389239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.389663 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.389664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.389728 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.390099 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.390103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.391715 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:54:12.391846 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.387851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:59823 (local address 127.20.199.62:38677)
0925 14:54:12.387889 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.387891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.387908 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:54:12.387930 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.387930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.387938 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.387979 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.388229 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.388671 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.389315 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.389575 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.389659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.391778 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:59823: BlockingRecv error: recv got EOF from 127.20.199.3:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:12.394066 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.386384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.20.199.62:38677)
0925 14:54:12.386532 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.386534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.386541 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:54:12.386563 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.386563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.386574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.386639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.387269 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.388009 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.388815 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.389026 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.389098 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.390522 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:12.390527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:12.390527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:12.390539 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:12.392485 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:12.392856 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:12.392859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:12.392859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:12.392954 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:12.393164 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:12.393172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:12.393172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:12.393389 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:12.393589 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:12.393834 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 14:54:12.393975 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:12 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:12.396713 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.392001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:48257)
0925 14:54:12.392128 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.392139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:12.392221 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:12.392375 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:12.392375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:12.392521 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:12.392624 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.392626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.393277 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.393278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.394114 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:12.394117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.394643 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.394644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:12.394702 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.395141 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:12.395145 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:12.396664 (+ 1519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
W20260925 14:54:12.396826 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:12.392052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:48257 (local address 127.20.199.62:38677)
0925 14:54:12.392105 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:12.392107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:12.392195 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:54:12.392296 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:12.392297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:12.392306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:12.392369 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:12.392715 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.393205 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.394245 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:12.394476 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:12.394588 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:12.396762 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:48257: BlockingRecv error: recv got EOF from 127.20.199.3:48257 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 14:54:12.396827 30429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:54:13 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:13 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:13.146891 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:13.142442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:44489)
0925 14:54:13.142580 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:13.142593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:13.142679 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:13.142850 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:13.142851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:13.143042 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:13.143089 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:13.143091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.143721 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.143721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:13.144257 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:13.144258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.144607 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.144607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:13.144645 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:13.145074 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:13.145078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:13.146836 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260925 14:54:13.147053 30436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:13.142485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:44489 (local address 127.20.199.62:38677)
0925 14:54:13.142599 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:13.142601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:13.142685 (+ 84us) server_negotiation.cc:407] Connection header received
0925 14:54:13.142747 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:13.142748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:13.142761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:13.142832 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:13.143157 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.143658 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.144342 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.144542 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.144597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:13.146966 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:44489: BlockingRecv error: recv got EOF from 127.20.199.1:44489 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
Sep 25 14:54:13 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:13 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:13.282343 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:13.278078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:49529)
0925 14:54:13.278236 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:13.278247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:13.278306 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:13.278423 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:13.278424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:13.278567 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:13.278614 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:13.278616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.279231 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.279231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:13.279752 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:13.279754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.280089 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.280090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:13.280127 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:13.280543 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:13.280547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:13.282277 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
W20260925 14:54:13.282410 30436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:13.278116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:49529 (local address 127.20.199.62:38677)
0925 14:54:13.278184 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:13.278187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:13.278305 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:54:13.278347 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:13.278348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:13.278359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:13.278407 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:13.278674 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.279168 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.279839 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.280025 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.280085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:13.282363 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:49529: BlockingRecv error: recv got EOF from 127.20.199.2:49529 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:54:13 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:13 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:13.401623 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:13.397181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:54423)
0925 14:54:13.397302 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:13.397314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:13.397369 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:13.397518 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:13.397518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:13.397672 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:13.397719 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:13.397721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.398418 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.398419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:13.398995 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:13.398996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.399336 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.399337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:13.399373 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:13.399811 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:13.399816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:13.401567 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
W20260925 14:54:13.401804 30436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:13.397302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:54423 (local address 127.20.199.62:38677)
0925 14:54:13.397383 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:13.397385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:13.397394 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:13.397436 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:13.397436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:13.397448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:13.397511 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:13.397821 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.398352 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.399074 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:13.399274 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:13.399330 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:13.401720 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:54423: BlockingRecv error: recv got EOF from 127.20.199.3:54423 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 14:54:14 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:14 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:14.151798 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:14.147304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:48039)
0925 14:54:14.147468 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:14.147479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:14.147536 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:14.147668 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:14.147668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:14.147822 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:14.147869 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:14.147870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.148512 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.148513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:14.149041 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:14.149042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.149418 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.149419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:14.149465 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:14.149934 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:14.149939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:14.151702 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260925 14:54:14.151844 30440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:14.147401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:48039 (local address 127.20.199.62:38677)
0925 14:54:14.147518 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:14.147520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:14.147529 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:14.147581 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:14.147582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:14.147593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:14.147649 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:14.147931 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.148425 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.149126 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.149329 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.149384 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:14.151775 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:48039: BlockingRecv error: recv got EOF from 127.20.199.1:48039 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
Sep 25 14:54:14 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:14 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:14.287237 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:14.282828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:60185)
0925 14:54:14.282964 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:14.282974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:14.283039 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:14.283178 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:14.283180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:14.283434 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:14.283496 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:14.283499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.284188 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.284189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:14.284718 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:14.284720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.285037 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.285038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:14.285076 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:14.285557 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:14.285561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:14.287183 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260925 14:54:14.287352 30440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:14.282891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:60185 (local address 127.20.199.62:38677)
0925 14:54:14.282967 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:14.282969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:14.283027 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:54:14.283085 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:14.283086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:14.283097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:14.283149 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:14.283580 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.284120 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.284792 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.284967 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.285015 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:14.287278 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:60185: BlockingRecv error: recv got EOF from 127.20.199.2:60185 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 14:54:14 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:14 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:14.406940 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:14.402049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:46783)
0925 14:54:14.402185 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:14.402200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:14.402261 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:14.402436 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:14.402437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:14.402724 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:14.402788 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:14.402790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.403464 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.403465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:14.404310 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:14.404313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.404627 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.404628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:14.404666 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:14.405074 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:14.405079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:14.406842 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
W20260925 14:54:14.406986 30440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:14.402219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:46783 (local address 127.20.199.62:38677)
0925 14:54:14.402290 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:14.402292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:14.402316 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:14.402338 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:14.402339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:14.402355 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:14.402406 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:14.402875 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.403360 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.404387 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:14.404566 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:14.404621 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:14.406921 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:46783: BlockingRecv error: recv got EOF from 127.20.199.3:46783 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 14:54:15 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:15 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:15.157202 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:15.152248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:55979)
0925 14:54:15.152395 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:15.152407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:15.152475 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:15.152666 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:15.152667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:15.152863 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:15.152912 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:15.152914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.153686 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.153687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:15.154225 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:15.154226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.154582 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.154583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:15.154626 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:15.155099 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:15.155104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:15.157080 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
W20260925 14:54:15.157260 30444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:15.152355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:55979 (local address 127.20.199.62:38677)
0925 14:54:15.152501 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:15.152503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:15.152510 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:54:15.152534 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:15.152534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:15.152546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:15.152614 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:15.152996 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.153586 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.154300 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.154515 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.154567 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:15.157208 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:55979: BlockingRecv error: recv got EOF from 127.20.199.1:55979 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
Sep 25 14:54:15 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:15 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:15.292364 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:15.287814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:41453)
0925 14:54:15.287986 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:15.287997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:15.288053 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:15.288174 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:15.288174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:15.288335 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:15.288382 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:15.288383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.288994 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.288995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:15.289552 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:15.289554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.289877 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.289878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:15.289915 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:15.290333 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:15.290338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:15.292307 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
W20260925 14:54:15.292454 30444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:15.287902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:41453 (local address 127.20.199.62:38677)
0925 14:54:15.287970 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:15.287972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:15.288050 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:54:15.288099 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:15.288099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:15.288110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:15.288168 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:15.288435 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.288925 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.289627 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.289813 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.289865 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:15.292390 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:41453: BlockingRecv error: recv got EOF from 127.20.199.2:41453 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 14:54:15 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:15 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:15.411885 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:15.407395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:55121)
0925 14:54:15.407590 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:15.407600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:15.407660 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:15.407792 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:15.407793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:15.407959 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:15.408006 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:15.408007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.408628 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.408629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:15.409163 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:15.409165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.409527 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.409528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:15.409570 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:15.409978 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:15.409982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:15.411830 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
W20260925 14:54:15.412015 30444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:15.407490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:55121 (local address 127.20.199.62:38677)
0925 14:54:15.407548 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:15.407550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:15.407665 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:54:15.407704 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:15.407705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:15.407718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:15.407773 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:15.408072 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.408556 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.409239 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:15.409456 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:15.409521 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:15.411934 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:55121: BlockingRecv error: recv got EOF from 127.20.199.3:55121 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 14:54:16 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:16 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:16.162477 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.157748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:35695)
0925 14:54:16.157865 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:16.157883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:16.157985 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:16.158192 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:16.158193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:16.158356 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:16.158405 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:16.158406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.159077 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.159078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:16.159701 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:16.159703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.160051 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.160052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:16.160099 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.160548 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:16.160553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:16.162378 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
W20260925 14:54:16.162513 30448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:16.157916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:35695 (local address 127.20.199.62:38677)
0925 14:54:16.158030 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:16.158032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:16.158040 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:16.158063 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:16.158064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:16.158082 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:16.158178 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:16.158498 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.158990 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.159793 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.159973 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.160018 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.162462 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:35695: BlockingRecv error: recv got EOF from 127.20.199.1:35695 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
Sep 25 14:54:16 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:16 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:16.304126 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.292908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:49135)
0925 14:54:16.293062 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:16.293074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:16.293153 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:16.293274 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:16.293275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:16.293467 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:16.293511 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:16.293513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.294204 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.294206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:16.294715 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:16.294717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.295065 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.295066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:16.295137 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.302280 (+ 7143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:16.302284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:16.304051 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
W20260925 14:54:16.304245 30448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:16.292984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:49135 (local address 127.20.199.62:38677)
0925 14:54:16.293053 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:16.293055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:16.293134 (+ 79us) server_negotiation.cc:407] Connection header received
0925 14:54:16.293207 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:16.293207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:16.293220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:16.293271 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:16.293588 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.294129 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.294796 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.294981 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.295057 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.304162 (+ 9105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:49135: BlockingRecv error: recv got EOF from 127.20.199.2:49135 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:54:16 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:16 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:16.417474 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.412372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:47737)
0925 14:54:16.412572 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:16.412583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:16.412678 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:16.412825 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:16.412826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:16.413013 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:16.413057 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:16.413058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.413765 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.413767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:16.414359 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:16.414361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.414793 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.414794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:16.414853 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.415355 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:16.415361 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:16.417336 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
W20260925 14:54:16.417488 30448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:16.412470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:47737 (local address 127.20.199.62:38677)
0925 14:54:16.412565 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:16.412567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:16.412660 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:54:16.412735 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:16.412736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:16.412756 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:16.412819 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:16.413137 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.413687 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.414484 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:16.414708 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:16.414792 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:16.417430 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:47737: BlockingRecv error: recv got EOF from 127.20.199.3:47737 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 14:54:17 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:17 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:17.167379 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.163030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:33389)
0925 14:54:17.163161 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.163172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:17.163234 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:17.163400 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:17.163400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:17.163567 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:17.163613 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.163615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.164296 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.164299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.164818 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.164819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.165208 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.165209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.165269 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.165692 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:17.165697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:17.167324 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":100,"threads_started":1}
W20260925 14:54:17.167523 30452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:17.163074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:33389 (local address 127.20.199.62:38677)
0925 14:54:17.163220 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:17.163223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:17.163232 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:17.163300 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:17.163300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:17.163312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:17.163394 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:17.163705 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.164210 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.164908 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.165127 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.165198 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.167430 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:33389: BlockingRecv error: recv got EOF from 127.20.199.1:33389 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
Sep 25 14:54:17 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:17 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:17.308902 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.304594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:48477)
0925 14:54:17.304748 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.304762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:17.304834 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:17.304952 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:17.304952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:17.305108 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:17.305155 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.305156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.305839 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.305839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.306374 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.306376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.306678 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.306679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.306715 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.307148 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:17.307152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:17.308847 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
W20260925 14:54:17.309032 30452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:17.304724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:48477 (local address 127.20.199.62:38677)
0925 14:54:17.304777 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:17.304778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:17.304820 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:54:17.304879 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:17.304880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:17.304889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:17.304936 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:17.305222 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.305779 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.306437 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.306611 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.306667 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.308953 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:48477: BlockingRecv error: recv got EOF from 127.20.199.2:48477 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:54:17 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:17 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:17.422679 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.418131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:36027)
0925 14:54:17.418319 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.418331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:17.418428 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:17.418578 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:17.418579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:17.418758 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:17.418803 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.418805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.419471 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.419473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.420001 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:17.420003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.420377 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.420378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:17.420435 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.420867 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:17.420871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:17.422609 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
W20260925 14:54:17.422771 30452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:17.418247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:36027 (local address 127.20.199.62:38677)
0925 14:54:17.418319 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:17.418321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:17.418409 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:54:17.418487 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:17.418488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:17.418509 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:17.418574 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:17.418888 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.419380 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.420087 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:17.420275 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:17.420379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:17.422702 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:36027: BlockingRecv error: recv got EOF from 127.20.199.3:36027 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 14:54:18 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:18 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:18.172622 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.167943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:39327)
0925 14:54:18.168076 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.168103 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:18.168169 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:18.168335 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:18.168336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:18.168492 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:18.168538 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.168539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.169231 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.169232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.169792 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.169794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.170189 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.170190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.170229 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.170684 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:18.170688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:18.172564 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
W20260925 14:54:18.172799 30456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:18.168002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:39327 (local address 127.20.199.62:38677)
0925 14:54:18.168152 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:18.168154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:18.168164 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:18.168242 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:18.168243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:18.168254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:18.168319 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:18.168627 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.169155 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.169891 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.170113 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.170183 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.172691 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:39327: BlockingRecv error: recv got EOF from 127.20.199.1:39327 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
Sep 25 14:54:18 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:18 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:18.314246 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.309392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:47703)
0925 14:54:18.309609 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.309625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:18.309684 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:18.309818 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:18.309819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:18.309985 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:18.310032 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.310034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.310642 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.310643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.311179 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.311181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.311699 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.311700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.311765 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.312201 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:18.312235 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:18.314150 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
W20260925 14:54:18.314293 30456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:18.309522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:47703 (local address 127.20.199.62:38677)
0925 14:54:18.309584 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:18.309586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:18.309684 (+ 98us) server_negotiation.cc:407] Connection header received
0925 14:54:18.309732 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:18.309733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:18.309745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:18.309802 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:18.310092 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.310567 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.311275 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.311600 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.311698 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.314230 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:47703: BlockingRecv error: recv got EOF from 127.20.199.2:47703 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 14:54:18 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:18 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:18.427634 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.423171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:60035)
0925 14:54:18.423309 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.423320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:18.423379 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:18.423512 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:18.423513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:18.423676 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:18.423726 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.423727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.424341 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.424341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.424851 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:18.424852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.425147 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.425148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:18.425189 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.425762 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:18.425767 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:18.427576 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260925 14:54:18.427793 30456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:18.423233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:60035 (local address 127.20.199.62:38677)
0925 14:54:18.423296 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:18.423298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:18.423374 (+ 76us) server_negotiation.cc:407] Connection header received
0925 14:54:18.423434 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:18.423434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:18.423446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:18.423494 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:18.423787 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.424278 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.424918 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:18.425082 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:18.425138 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:18.427718 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:60035: BlockingRecv error: recv got EOF from 127.20.199.3:60035 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:54:19 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:19 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:19.177884 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.173239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:59911)
0925 14:54:19.173419 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.173447 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.173522 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.173709 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.173710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.173888 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:19.173937 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.173939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.174559 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.174560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.175158 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.175159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.175497 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.175498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.175543 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.175965 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:19.175971 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:19.177826 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
W20260925 14:54:19.177985 30460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:19.173339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59911 (local address 127.20.199.62:38677)
0925 14:54:19.173473 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.173475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.173496 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:54:19.173583 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.173584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.173595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.173652 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.174008 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.174490 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.175228 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.175412 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.175461 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.177913 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:59911: BlockingRecv error: recv got EOF from 127.20.199.1:59911 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":98,"threads_started":1}
Sep 25 14:54:19 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:19 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:19.319363 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.314750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:53483)
0925 14:54:19.314910 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.314921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.315010 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.315174 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.315174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.315356 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:19.315403 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.315405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.316101 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.316102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.316656 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.316657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.317030 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.317031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.317088 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.317607 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:19.317612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:19.319307 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
W20260925 14:54:19.319496 30460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:19.314824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:53483 (local address 127.20.199.62:38677)
0925 14:54:19.314907 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.314909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.314989 (+ 80us) server_negotiation.cc:407] Connection header received
0925 14:54:19.315075 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.315077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.315092 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.315171 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.315511 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.316017 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.316756 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.316957 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.317033 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.319418 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:53483: BlockingRecv error: recv got EOF from 127.20.199.2:53483 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 14:54:19 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:19 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:19.432647 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.428145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:55187)
0925 14:54:19.428284 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.428295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:19.428365 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:19.428505 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:19.428506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:19.428663 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:19.428709 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.428711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.429482 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.429483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.430009 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:19.430010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.430380 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.430382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:19.430441 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.430875 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:19.430879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:19.432579 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260925 14:54:19.432802 30460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:19.428232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:55187 (local address 127.20.199.62:38677)
0925 14:54:19.428303 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:19.428305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:19.428342 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:54:19.428427 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:19.428427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:19.428440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:19.428491 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:19.428772 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.429377 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.430093 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:19.430293 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:19.430379 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:19.432669 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:55187: BlockingRecv error: recv got EOF from 127.20.199.3:55187 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:54:20 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:20 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:20.183007 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.178416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:35859)
0925 14:54:20.178572 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:20.178586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:20.178654 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:20.178833 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:20.178834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:20.179002 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:20.179048 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.179050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.179694 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.179695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.180218 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.180219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.180593 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.180594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.180635 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.181156 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:20.181162 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:20.182952 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
W20260925 14:54:20.183164 30464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:20.178554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:35859 (local address 127.20.199.62:38677)
0925 14:54:20.178714 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:20.178716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:20.178723 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:54:20.178744 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:20.178745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:20.178755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:20.178813 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:20.179115 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.179629 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.180309 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.180525 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.180586 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.183074 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:35859: BlockingRecv error: recv got EOF from 127.20.199.1:35859 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
Sep 25 14:54:20 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:20 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:20.324666 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.319795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:57895)
0925 14:54:20.319982 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:20.319992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:20.320050 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:20.320192 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:20.320192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:20.320382 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:20.320431 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.320433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.321222 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.321223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.321804 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.321806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.322115 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.322116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.322167 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.322627 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:20.322631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:20.324587 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
W20260925 14:54:20.324848 30464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:20.319877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57895 (local address 127.20.199.62:38677)
0925 14:54:20.319969 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:20.319971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:20.320053 (+ 82us) server_negotiation.cc:407] Connection header received
0925 14:54:20.320096 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:20.320096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:20.320107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:20.320171 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:20.320494 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.321155 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.321873 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.322049 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.322098 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.324747 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:57895: BlockingRecv error: recv got EOF from 127.20.199.2:57895 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 14:54:20 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:20 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:20.437572 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.433142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:48573)
0925 14:54:20.433267 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:20.433280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:20.433351 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:20.433511 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:20.433512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:20.433666 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:20.433712 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.433714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.434322 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.434323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.434850 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.434852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.435174 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.435175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.435227 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.435697 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:20.435702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:20.437514 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260925 14:54:20.437667 30464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:20.433221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:48573 (local address 127.20.199.62:38677)
0925 14:54:20.433299 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:20.433300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:20.433335 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:54:20.433422 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:20.433423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:20.433435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:20.433482 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:20.433773 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.434259 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.434917 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.435089 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.435145 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.437600 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:48573: BlockingRecv error: recv got EOF from 127.20.199.3:48573 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 14:54:21 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:21 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:21.188264 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.183511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:46763)
0925 14:54:21.183662 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.183675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.183733 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.183903 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.183904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.184061 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.184108 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.184109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.184773 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.184773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.185313 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.185314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.185725 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.185727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.185776 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.186307 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:21.186311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.188196 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260925 14:54:21.188371 30468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:21.183586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:46763 (local address 127.20.199.62:38677)
0925 14:54:21.183715 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.183717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.183726 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:21.183808 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.183808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.183823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.183887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.184200 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.184702 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.185415 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.185636 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.185694 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.188296 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:46763: BlockingRecv error: recv got EOF from 127.20.199.1:46763 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
Sep 25 14:54:21 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:21 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:21.332921 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.325078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:52169)
0925 14:54:21.325236 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.325257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.325325 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.325538 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.325540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.325715 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.325769 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.325771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.326418 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.326419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.326958 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.326960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.327274 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.327274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.327313 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.330986 (+ 3673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:21.330990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.332817 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
W20260925 14:54:21.332973 30468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:21.325193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:52169 (local address 127.20.199.62:38677)
0925 14:54:21.325261 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.325263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.325318 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:54:21.325380 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.325380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.325391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.325491 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.325844 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.326333 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.327036 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.327213 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.327261 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.332903 (+ 5642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:52169: BlockingRecv error: recv got EOF from 127.20.199.2:52169 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 14:54:21 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:21 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:21.443349 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.438125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:40977)
0925 14:54:21.438358 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.438371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.438504 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.438676 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.438677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.438866 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.438911 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.438912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.439746 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.439747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.440346 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.440348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.440696 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.440697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.440737 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.441192 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:21.441197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.443266 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
W20260925 14:54:21.443380 30468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:21.438244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:40977 (local address 127.20.199.62:38677)
0925 14:54:21.438343 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.438346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.438483 (+ 137us) server_negotiation.cc:407] Connection header received
0925 14:54:21.438565 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.438566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.438583 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.438671 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.438994 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.439647 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.440422 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.440623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.440680 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.443332 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:40977: BlockingRecv error: recv got EOF from 127.20.199.3:40977 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 14:54:22 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:22 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:22.193872 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.188881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:51461)
0925 14:54:22.189056 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:22.189067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:22.189153 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:22.189346 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:22.189347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:22.189598 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:22.189660 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.189662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.190288 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.190289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.190793 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.190794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.191145 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.191146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.191244 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.191667 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:22.191671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:22.193809 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
W20260925 14:54:22.193944 30472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:22.188986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:51461 (local address 127.20.199.62:38677)
0925 14:54:22.189097 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:22.189099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:22.189108 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:22.189242 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:22.189243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:22.189257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:22.189323 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:22.189729 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.190225 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.190868 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.191080 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.191132 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.193891 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:51461: BlockingRecv error: recv got EOF from 127.20.199.1:51461 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
Sep 25 14:54:22 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:22 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:22.337743 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.333353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:52503)
0925 14:54:22.333529 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:22.333539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:22.333611 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:22.333733 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:22.333734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:22.333885 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:22.333931 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.333932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.334519 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.334520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.335022 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.335024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.335312 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.335313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.335349 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.335840 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:22.335844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:22.337621 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
W20260925 14:54:22.337781 30472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:22.333452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:52503 (local address 127.20.199.62:38677)
0925 14:54:22.333522 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:22.333524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:22.333611 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:54:22.333662 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:22.333662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:22.333673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:22.333720 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:22.333987 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.334452 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.335087 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.335255 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.335300 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.337720 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:52503: BlockingRecv error: recv got EOF from 127.20.199.2:52503 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:54:22 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:22 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:22.448338 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.443873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:50815)
0925 14:54:22.444017 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:22.444035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:22.444089 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:22.444246 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:22.444246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:22.444404 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:22.444455 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.444456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.445080 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.445082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.445719 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.445722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.446040 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.446041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.446082 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.446542 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:22.446547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:22.448277 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260925 14:54:22.448410 30472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:22.443981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:50815 (local address 127.20.199.62:38677)
0925 14:54:22.444050 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:22.444052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:22.444076 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:22.444145 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:22.444146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:22.444159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:22.444217 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:22.444523 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.444995 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.445799 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.445975 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.446025 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.448362 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:50815: BlockingRecv error: recv got EOF from 127.20.199.3:50815 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:54:23 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:23 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:23.198967 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.194345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:59421)
0925 14:54:23.194488 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.194498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.194550 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.194697 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.194698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.194860 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:23.194906 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.194907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.195651 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.195652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.196157 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.196158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.196504 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.196505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.196542 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.197049 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:23.197053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:23.198850 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
W20260925 14:54:23.199050 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:23.194422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59421 (local address 127.20.199.62:38677)
0925 14:54:23.194563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.194564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.194590 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:54:23.194619 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.194620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.194631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.194681 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.194975 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.195581 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.196228 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.196442 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.196493 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.198943 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:59421: BlockingRecv error: recv got EOF from 127.20.199.1:59421 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
Sep 25 14:54:23 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:23 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:23.343149 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.338298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:52865)
0925 14:54:23.338444 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.338455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.338528 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.338663 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.338664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.338839 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:23.338882 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.338884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.339549 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.339550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.340090 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.340092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.340468 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.340469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.340534 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.341122 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:23.341126 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:23.343045 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
W20260925 14:54:23.343202 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:23.338457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:52865 (local address 127.20.199.62:38677)
0925 14:54:23.338532 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.338535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.338560 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:54:23.338587 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.338588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.338599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.338647 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.338963 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.339463 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.340181 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.340382 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.340467 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.343128 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:52865: BlockingRecv error: recv got EOF from 127.20.199.2:52865 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 14:54:23 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:23 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:23.453822 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.448896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:59661)
0925 14:54:23.449049 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.449070 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.449161 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.449344 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.449345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.449570 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:23.449612 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.449613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.450306 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.450306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.450845 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.450847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.451200 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.451201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.451258 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.451735 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:23.451739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:23.453756 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":52,"threads_started":1}
W20260925 14:54:23.453939 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:23.449009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:59661 (local address 127.20.199.62:38677)
0925 14:54:23.449084 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.449086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.449146 (+ 60us) server_negotiation.cc:407] Connection header received
0925 14:54:23.449250 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.449251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.449263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.449335 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.449713 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.450206 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.450931 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.451117 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.451203 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.453857 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:59661: BlockingRecv error: recv got EOF from 127.20.199.3:59661 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 14:54:24 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:24 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:24.204345 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.199554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:58965)
0925 14:54:24.199723 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.199734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.199788 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.199917 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.199918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.200089 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:24.200135 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.200137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.200773 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.200774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.201514 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.201517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.201904 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.201905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.201945 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.202367 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:24.202371 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:24.204243 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
W20260925 14:54:24.204403 30480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:24.199664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58965 (local address 127.20.199.62:38677)
0925 14:54:24.199794 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.199795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.199804 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:24.199829 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.199829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.199840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.199899 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.200204 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.200708 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.201571 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.201842 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.201897 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.204327 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:58965: BlockingRecv error: recv got EOF from 127.20.199.1:58965 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
Sep 25 14:54:24 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:24 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:24.347998 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.343610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:57343)
0925 14:54:24.343747 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.343757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.343809 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.343941 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.343941 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.344100 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:24.344143 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.344145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.344780 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.344780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.345297 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.345298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.345676 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.345677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.345719 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.346230 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:24.346234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:24.347934 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260925 14:54:24.348138 30480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:24.343687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57343 (local address 127.20.199.62:38677)
0925 14:54:24.343755 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.343757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.343797 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:54:24.343865 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.343866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.343877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.343925 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.344210 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.344710 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.345376 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.345602 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.345662 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.348057 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:57343: BlockingRecv error: recv got EOF from 127.20.199.2:57343 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:54:24 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:24 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:24.459288 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.454361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:44109)
0925 14:54:24.454620 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.454632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.454702 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.454832 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.454833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.455011 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:24.455072 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.455074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.455748 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.455748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.456307 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.456308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.456721 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.456724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.456774 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.457346 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:24.457356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:24.459233 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
W20260925 14:54:24.459434 30480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:24.454435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:44109 (local address 127.20.199.62:38677)
0925 14:54:24.454638 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.454640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.454703 (+ 63us) server_negotiation.cc:407] Connection header received
0925 14:54:24.454746 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.454747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.454760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.454814 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.455138 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.455678 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.456432 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.456623 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.456688 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.459348 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:44109: BlockingRecv error: recv got EOF from 127.20.199.3:44109 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
Sep 25 14:54:25 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:25 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:25.209270 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.204874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:36695)
0925 14:54:25.205041 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:25.205052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:25.205107 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:25.205256 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:25.205257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:25.205442 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:25.205487 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.205488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.206100 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.206100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.206609 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.206610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.206940 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.206941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.206980 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.207468 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:25.207473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:25.209214 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
W20260925 14:54:25.209586 30484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:25.204994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:36695 (local address 127.20.199.62:38677)
0925 14:54:25.205129 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:25.205131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:25.205139 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:25.205161 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:25.205162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:25.205174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:25.205240 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:25.205543 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.206036 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.206687 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.206877 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.206932 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.209485 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:36695: BlockingRecv error: recv got EOF from 127.20.199.1:36695 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
Sep 25 14:54:25 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:25 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:25.353179 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.348459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:33751)
0925 14:54:25.348605 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:25.348620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:25.348676 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:25.348862 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:25.348863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:25.349024 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:25.349069 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.349071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.349813 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.349813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.350322 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.350323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.350680 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.350680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.350718 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.351199 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:25.351204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:25.353118 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260925 14:54:25.353333 30484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:25.348649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:33751 (local address 127.20.199.62:38677)
0925 14:54:25.348723 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:25.348725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:25.348735 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:25.348777 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:25.348778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:25.348789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:25.348848 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:25.349139 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.349740 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.350391 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.350609 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.350666 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.353246 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:33751: BlockingRecv error: recv got EOF from 127.20.199.2:33751 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:54:25 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:25 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:25.464437 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.459700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:59099)
0925 14:54:25.459838 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:25.459848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:25.459894 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:25.460034 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:25.460035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:25.460194 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:25.460240 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.460241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.460918 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.460919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.461740 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.461744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.462078 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.462079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.462117 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.462578 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:25.462590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:25.464381 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260925 14:54:25.464506 30484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:25.459801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:59099 (local address 127.20.199.62:38677)
0925 14:54:25.459866 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:25.459868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:25.459879 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:25.459950 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:25.459950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:25.459962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:25.460019 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:25.460330 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.460851 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.461814 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.462016 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.462070 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.464462 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:59099: BlockingRecv error: recv got EOF from 127.20.199.3:59099 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 14:54:26 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:26 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:26.214497 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.209707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:54415)
0925 14:54:26.209993 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.210006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.210077 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.210215 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.210216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.210386 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:26.210445 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.210446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.211095 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.211096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.211671 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.211673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.212036 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.212037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.212086 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.212597 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:26.212602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:26.214440 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":192,"threads_started":1}
W20260925 14:54:26.214627 30488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:26.209887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:54415 (local address 127.20.199.62:38677)
0925 14:54:26.210022 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.210025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.210059 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:54:26.210140 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.210140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.210151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.210209 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.210508 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.211009 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.211758 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.211949 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.212016 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.214534 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:54415: BlockingRecv error: recv got EOF from 127.20.199.1:54415 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":52,"threads_started":1}
Sep 25 14:54:26 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:26 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:26.366043 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.353740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:55267)
0925 14:54:26.353902 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.353915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.353998 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.354109 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.354110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.354284 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:26.354330 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.354331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.354938 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.354939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.355471 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.355472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.355795 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.355796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.355846 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.364092 (+ 8246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:26.364098 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:26.365974 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260925 14:54:26.366166 30488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:26.353852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:55267 (local address 127.20.199.62:38677)
0925 14:54:26.353903 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.353905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.353981 (+ 76us) server_negotiation.cc:407] Connection header received
0925 14:54:26.354046 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.354046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.354056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.354098 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.354390 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.354871 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.355537 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.355715 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.355763 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.366059 (+ 10296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:55267: BlockingRecv error: recv got EOF from 127.20.199.2:55267 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 14:54:26 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:26 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.3@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:26.469803 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.464932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.3:53851)
0925 14:54:26.465131 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.465142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.465237 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.465390 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.465390 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.465592 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:26.465634 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.465636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.466309 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.466310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.466833 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.466835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.467200 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.467201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.467266 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.467740 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:26.467747 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:26.469746 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
W20260925 14:54:26.469928 30488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:26.465040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:53851 (local address 127.20.199.62:38677)
0925 14:54:26.465125 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.465127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.465218 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:54:26.465293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.465294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.465309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.465384 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.465735 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.466230 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.466916 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.467109 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.467198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.469831 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.3:53851: BlockingRecv error: recv got EOF from 127.20.199.3:53851 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 14:54:27 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:27 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.1@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:27.219981 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.215107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.1:40569)
0925 14:54:27.215371 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.215386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.215507 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.215744 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.215745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.215955 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.216005 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.216007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.216663 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.216664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.217225 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.217227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.217591 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.217592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.217642 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.218076 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:27.218081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.219918 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
W20260925 14:54:27.220081 30492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:27.215187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:40569 (local address 127.20.199.62:38677)
0925 14:54:27.215317 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.215319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.215507 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:54:27.215625 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.215626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.215639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.215715 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.216080 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.216567 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.217304 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.217506 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.217556 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.220009 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.1:40569: BlockingRecv error: recv got EOF from 127.20.199.1:40569 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
Sep 25 14:54:27 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:54:27 dist-test-slave-8qt6 krb5kdc[29431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.199.2@KRBTEST.COM for oryx/127.20.199.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:54:27.371337 30493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.366582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38677 (local address 127.20.199.2:46917)
0925 14:54:27.366737 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.366748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.366802 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.367011 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.367011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.367262 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.367339 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.367342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.368085 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.368086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.368593 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.368595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.368938 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.368939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.368977 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.369506 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:27.369511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.371280 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.199.62:38677: Server oryx/127.20.199.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
W20260925 14:54:27.371552 30492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:27.366657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:46917 (local address 127.20.199.62:38677)
0925 14:54:27.366764 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.366766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.366793 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:54:27.366865 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.366866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.366896 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.366987 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.367421 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.368005 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.368676 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.368867 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.368931 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.371475 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.199.2:46917: BlockingRecv error: recv got EOF from 127.20.199.2:46917 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 14:54:27.388764 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30014
I20260925 14:54:27.393455 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30167
I20260925 14:54:27.397660 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30300
I20260925 14:54:27.401909 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 29944
2026-09-25T14:54:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17423 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:27Z Disabled control of system clock
I20260925 14:54:27.428879 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40487
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:54:27.510381 30497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.510583 30497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.510603 30497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.511729 30497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:27.511765 30497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.511777 30497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.511790 30497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:27.511802 30497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:27.513170 30497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:40487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:27.513378 30497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.513705 30497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.518643 30503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.518682 30497 server_base.cc:1034] running on GCE node
W20260925 14:54:27.518694 30505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.518651 30502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.519048 30497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.519326 30497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.520462 30497 hybrid_clock.cc:648] HybridClock initialized: now 1790348067520450 us; error 29 us; skew 500 ppm
I20260925 14:54:27.521767 30497 webserver.cc:460] Webserver started at http://127.20.199.62:42251/ using document root <none> and password file <none>
I20260925 14:54:27.522012 30497 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.522060 30497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.522238 30497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.523308 30497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "a1358d2a8c3d49bf832e325e296f7ca6"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "8e4b34b19b485128fca8eb4e905e0937"
server_key_iv: "b4661a967feaaa45cf149a87c1bb2a79"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.523599 30497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "a1358d2a8c3d49bf832e325e296f7ca6"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "8e4b34b19b485128fca8eb4e905e0937"
server_key_iv: "b4661a967feaaa45cf149a87c1bb2a79"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.524789 30497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:27.525523 30511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.525739 30497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:27.525828 30497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "a1358d2a8c3d49bf832e325e296f7ca6"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "8e4b34b19b485128fca8eb4e905e0937"
server_key_iv: "b4661a967feaaa45cf149a87c1bb2a79"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.525897 30497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.537755 30497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.538046 30497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.538184 30497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.542349 30497 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:40487
I20260925 14:54:27.542374 30563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:40487 every 8 connection(s)
I20260925 14:54:27.542721 30497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:27.543351 30564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:27.545912 30564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6: Bootstrap starting.
I20260925 14:54:27.546486 30564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:27.546772 30564 log.cc:824] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:27.547421 30564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6: No bootstrap required, opened a new log
I20260925 14:54:27.548938 30564 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40487 } }
I20260925 14:54:27.549077 30564 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:27.549099 30564 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1358d2a8c3d49bf832e325e296f7ca6, State: Initialized, Role: FOLLOWER
I20260925 14:54:27.549283 30564 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40487 } }
I20260925 14:54:27.549371 30564 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:27.549435 30564 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:27.549489 30564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:27.549993 30564 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40487 } }
I20260925 14:54:27.550099 30564 leader_election.cc:302] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1358d2a8c3d49bf832e325e296f7ca6; no voters:
I20260925 14:54:27.550274 30564 leader_election.cc:288] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:27.550340 30567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:27.550536 30567 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [term 1 LEADER]: Becoming Leader. State: Replica: a1358d2a8c3d49bf832e325e296f7ca6, State: Running, Role: LEADER
I20260925 14:54:27.550540 30564 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:27.550640 30567 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40487 } }
I20260925 14:54:27.550992 30569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1358d2a8c3d49bf832e325e296f7ca6. Latest consensus state: current_term: 1 leader_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40487 } } }
I20260925 14:54:27.551119 30568 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1358d2a8c3d49bf832e325e296f7ca6" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40487 } } }
I20260925 14:54:27.551160 30569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:27.551235 30568 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:27.551584 30576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:27.552075 30576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:27.552894 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30497
I20260925 14:54:27.552996 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:54:27.553215 21276 external_mini_cluster.cc:1428] Setting key a4611e9bb1627b02d682c164ba74231d
I20260925 14:54:27.553885 30576 catalog_manager.cc:1347] Generated new cluster ID: 18f92cf568e64f6c992b1db18e41c9dc
I20260925 14:54:27.553944 30576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:27.563259 30576 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:27.563939 30576 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:27.568378 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.553694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.20.199.62:40487)
0925 14:54:27.553866 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.553875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.563187 (+ 9312us) server_negotiation.cc:407] Connection header received
0925 14:54:27.563321 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.563329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.563403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.563562 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.563780 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.565151 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.566267 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.566874 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.566982 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.567701 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.567705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.567706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.567908 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.567947 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.567960 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.568018 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.568097 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.568190 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260925 14:54:27.584188 30576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1358d2a8c3d49bf832e325e296f7ca6: Generated new TSK 0
I20260925 14:54:27.584388 30576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:27.587455 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40487
--builtin_ntp_servers=127.20.199.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:54:27.668040 30588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.668239 30588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.668259 30588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.669484 30588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.669524 30588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.669550 30588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:27.670859 30588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:27.671101 30588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.671396 30588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.677199 30596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.677182 30594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.677229 30588 server_base.cc:1034] running on GCE node
W20260925 14:54:27.677215 30593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.677641 30588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.677917 30588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.679104 30588 hybrid_clock.cc:648] HybridClock initialized: now 1790348067679095 us; error 35 us; skew 500 ppm
I20260925 14:54:27.680404 30588 webserver.cc:460] Webserver started at http://127.20.199.1:40153/ using document root <none> and password file <none>
I20260925 14:54:27.680660 30588 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.680730 30588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.680881 30588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.682086 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "ba6089594dbe467ab9562d984950d43e"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "41318001f8f66f64db069e3b4c571888"
server_key_iv: "687446cf7ed27ece79db097a855ed8dd"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.682452 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "ba6089594dbe467ab9562d984950d43e"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "41318001f8f66f64db069e3b4c571888"
server_key_iv: "687446cf7ed27ece79db097a855ed8dd"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.683732 30588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:27.684440 30602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.684612 30588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:27.684716 30588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "ba6089594dbe467ab9562d984950d43e"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "41318001f8f66f64db069e3b4c571888"
server_key_iv: "687446cf7ed27ece79db097a855ed8dd"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.684789 30588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.694641 30588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.694940 30588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.695096 30588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.695317 30588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:27.695636 30588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:27.695689 30588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.695724 30588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:27.695752 30588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.701350 30588 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:37801
I20260925 14:54:27.701426 30715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:37801 every 8 connection(s)
I20260925 14:54:27.701735 30588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:27.702088 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30588
I20260925 14:54:27.702188 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:27.702397 21276 external_mini_cluster.cc:1428] Setting key 6b1baa2bd2dc454ef12cb411667d32a2
I20260925 14:54:27.703480 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40487
--builtin_ntp_servers=127.20.199.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:54:27.708176 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.702569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:57085 (local address 127.20.199.62:40487)
0925 14:54:27.702617 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.702620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.702897 (+ 277us) server_negotiation.cc:407] Connection header received
0925 14:54:27.703616 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.703618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.703633 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.703706 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.704639 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.705178 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.706723 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.706915 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.706988 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.707913 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.707914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.707914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.707919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.707934 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.707937 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.707991 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.708084 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.708117 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:27.708477 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.702542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40487 (local address 127.20.199.1:57085)
0925 14:54:27.702777 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.702820 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.703475 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.703791 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.703795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.703812 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.704522 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.704529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.705276 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.705277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.706623 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.706626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.707004 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.707006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.707171 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.707740 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:27.707741 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.707782 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:27.707814 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:27.707998 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:27.708003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:27.708007 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.708090 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.708211 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260925 14:54:27.709092 30716 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40487
I20260925 14:54:27.709213 30716 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:27.709462 30716 heartbeater.cc:511] Master 127.20.199.62:40487 requested a full tablet report, sending...
I20260925 14:54:27.710037 30528 ts_manager.cc:194] Registered new tserver with Master: ba6089594dbe467ab9562d984950d43e (127.20.199.1:37801)
I20260925 14:54:27.710680 30528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.1:57085
W20260925 14:54:27.784413 30719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.784605 30719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.784626 30719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.785842 30719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.785881 30719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.785926 30719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:27.787241 30719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:27.787500 30719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.787786 30719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.792917 30725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.792984 30724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.792902 30727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.792980 30719 server_base.cc:1034] running on GCE node
I20260925 14:54:27.793357 30719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.793627 30719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.794816 30719 hybrid_clock.cc:648] HybridClock initialized: now 1790348067794790 us; error 36 us; skew 500 ppm
I20260925 14:54:27.795991 30719 webserver.cc:460] Webserver started at http://127.20.199.2:46221/ using document root <none> and password file <none>
I20260925 14:54:27.796247 30719 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.796314 30719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.796468 30719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.797582 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "82f087d894c5446293529697aeb8125a"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "97298c4505c9dc5e1d0d7e39f221d3a3"
server_key_iv: "721344a431a24cbd2d3d4432358f28e5"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.797948 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "82f087d894c5446293529697aeb8125a"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "97298c4505c9dc5e1d0d7e39f221d3a3"
server_key_iv: "721344a431a24cbd2d3d4432358f28e5"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.799331 30719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:54:27.800032 30733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.800199 30719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:27.800294 30719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "82f087d894c5446293529697aeb8125a"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "97298c4505c9dc5e1d0d7e39f221d3a3"
server_key_iv: "721344a431a24cbd2d3d4432358f28e5"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.800367 30719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.814498 30719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.814791 30719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.814937 30719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.815155 30719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:27.815475 30719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:27.815528 30719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.815568 30719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:27.815596 30719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.821306 30719 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:45629
I20260925 14:54:27.821354 30846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:45629 every 8 connection(s)
I20260925 14:54:27.821677 30719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:27.826874 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.822376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:49255 (local address 127.20.199.62:40487)
0925 14:54:27.822421 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.822423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.822647 (+ 224us) server_negotiation.cc:407] Connection header received
0925 14:54:27.823204 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.823205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.823227 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.823277 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.823965 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.824472 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.825495 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.825702 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.825789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.826564 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.826565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.826566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.826571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.826587 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.826599 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.826659 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.826754 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.826791 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:27.827083 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.822344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40487 (local address 127.20.199.2:49255)
0925 14:54:27.822524 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.822558 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.823077 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.823358 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.823365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.823378 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.823865 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.823870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.824567 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.824568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.825356 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.825358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.825797 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.825799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.825934 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.826416 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:27.826416 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.826444 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:27.826466 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:27.826660 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:27.826667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:27.826670 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.826757 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.826859 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260925 14:54:27.827546 30847 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40487
I20260925 14:54:27.827656 30847 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:27.827841 30847 heartbeater.cc:511] Master 127.20.199.62:40487 requested a full tablet report, sending...
I20260925 14:54:27.828227 30528 ts_manager.cc:194] Registered new tserver with Master: 82f087d894c5446293529697aeb8125a (127.20.199.2:45629)
I20260925 14:54:27.828649 30528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.2:49255
I20260925 14:54:27.828926 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30719
I20260925 14:54:27.829025 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:27.829250 21276 external_mini_cluster.cc:1428] Setting key bd03a66f2fe3f67437275413d80bf989
I20260925 14:54:27.830154 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40487
--builtin_ntp_servers=127.20.199.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:54:27.909094 30850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.909309 30850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.909330 30850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.910461 30850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.910501 30850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.910524 30850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:27.911738 30850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:27.911904 30850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.912122 30850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:54:27.917006 30850 server_base.cc:1034] running on GCE node
W20260925 14:54:27.916983 30858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.916978 30856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.916961 30855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.917333 30850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.917611 30850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.918785 30850 hybrid_clock.cc:648] HybridClock initialized: now 1790348067918773 us; error 24 us; skew 500 ppm
I20260925 14:54:27.920008 30850 webserver.cc:460] Webserver started at http://127.20.199.3:34221/ using document root <none> and password file <none>
I20260925 14:54:27.920240 30850 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.920310 30850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.920460 30850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.921571 30850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "9607a56deed343ceb88d9b489a1bbd67"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "fdb3368d242ed3883f9065fa8fc43c69"
server_key_iv: "5f378f017a16e79f69e86c8dc07db015"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.921942 30850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "9607a56deed343ceb88d9b489a1bbd67"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "fdb3368d242ed3883f9065fa8fc43c69"
server_key_iv: "5f378f017a16e79f69e86c8dc07db015"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.923182 30850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:27.923843 30864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.924003 30850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.924095 30850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "9607a56deed343ceb88d9b489a1bbd67"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-8qt6"
server_key: "fdb3368d242ed3883f9065fa8fc43c69"
server_key_iv: "5f378f017a16e79f69e86c8dc07db015"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.924167 30850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.935457 30850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.935751 30850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.935896 30850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.936100 30850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:27.936463 30850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:27.936517 30850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.936563 30850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:27.936589 30850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.943095 30850 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:40821
I20260925 14:54:27.943138 30977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:40821 every 8 connection(s)
I20260925 14:54:27.943444 30850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:27.944489 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30850
I20260925 14:54:27.944583 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:27.944780 21276 external_mini_cluster.cc:1428] Setting key d7991ca70e04f9a215ba4fd0a5ee1643
I20260925 14:54:27.948839 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.944105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:40125 (local address 127.20.199.62:40487)
0925 14:54:27.944168 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.944170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.944402 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:54:27.945103 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.945104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.945116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.945179 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.945971 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.946471 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.947449 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.947658 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.947741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.948586 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.948587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.948588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.948592 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.948607 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.948611 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.948659 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.948736 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.948762 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:54:27.949050 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.944060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40487 (local address 127.20.199.3:40125)
0925 14:54:27.944234 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.944274 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.944994 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.945249 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.945253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.945264 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.945883 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.945888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.946571 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.946572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.947346 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.947348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.947750 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.947751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.947895 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.948437 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:27.948437 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.948465 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:27.948485 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:27.948660 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:27.948665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:27.948668 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.948739 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.948835 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20260925 14:54:27.949515 30978 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40487
I20260925 14:54:27.949594 30978 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:27.949786 30978 heartbeater.cc:511] Master 127.20.199.62:40487 requested a full tablet report, sending...
I20260925 14:54:27.950139 30528 ts_manager.cc:194] Registered new tserver with Master: 9607a56deed343ceb88d9b489a1bbd67 (127.20.199.3:40821)
I20260925 14:54:27.950565 30528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.3:40125
I20260925 14:54:27.956470 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:54:27.958156 30587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:27.957766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.20.199.62:40487)
0925 14:54:27.957810 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.957812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.957898 (+ 86us) server_negotiation.cc:407] Connection header received
0925 14:54:27.957945 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.957946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.957953 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.958000 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.958112 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34282: BlockingRecv error: recv got EOF from 127.0.0.1:34282 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:54:27.958314 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30588
I20260925 14:54:27.962833 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30719
I20260925 14:54:27.967089 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30850
I20260925 14:54:27.971342 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30497
2026-09-25T14:54:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (568 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:27Z Disabled control of system clock
I20260925 14:54:27.997581 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:44929
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:54:28.076175 30989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:28.076414 30989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:28.076448 30989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:28.077625 30989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:28.077688 30989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:28.077709 30989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:28.077730 30989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:28.077762 30989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:28.079100 30989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:44929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:28.079376 30989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:28.079665 30989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:28.084602 30994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:28.084609 30989 server_base.cc:1034] running on GCE node
W20260925 14:54:28.084579 30997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:28.084591 30995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:28.085013 30989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:28.085310 30989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:28.086472 30989 hybrid_clock.cc:648] HybridClock initialized: now 1790348068086452 us; error 65 us; skew 500 ppm
I20260925 14:54:28.087752 30989 webserver.cc:460] Webserver started at http://127.20.199.62:46385/ using document root <none> and password file <none>
I20260925 14:54:28.088001 30989 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:28.088071 30989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:28.088220 30989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:28.089283 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "458139ef94d648ad8199f82098a14b2f"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "ab5c48ceb2ceab0a58286cd912f3e185"
server_key_iv: "79dad7ec87d34b42c8c6a279533df3f2"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.089701 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "458139ef94d648ad8199f82098a14b2f"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "ab5c48ceb2ceab0a58286cd912f3e185"
server_key_iv: "79dad7ec87d34b42c8c6a279533df3f2"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.091006 30989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:28.091850 31003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.092069 30989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:28.092157 30989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "458139ef94d648ad8199f82098a14b2f"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "ab5c48ceb2ceab0a58286cd912f3e185"
server_key_iv: "79dad7ec87d34b42c8c6a279533df3f2"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.092240 30989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:28.095702 30989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:28.095966 30989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:28.096118 30989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:28.100064 30989 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:44929
I20260925 14:54:28.100096 31055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:44929 every 8 connection(s)
I20260925 14:54:28.100448 30989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:28.101142 31056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:28.101487 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 30989
I20260925 14:54:28.101576 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:54:28.101780 21276 external_mini_cluster.cc:1428] Setting key 817662e498e48120720246f338d9cbaf
I20260925 14:54:28.104305 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.102364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.20.199.62:44929)
0925 14:54:28.102569 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:28.102579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:28.102611 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:54:28.103030 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:28.103038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:28.103093 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:28.103244 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:28.103850 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:28.103854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:28.103854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:28.103859 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:28.103905 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:28.103920 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:28.104030 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:28.104068 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:54:28.104119 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260925 14:54:28.105340 31056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f: Bootstrap starting.
I20260925 14:54:28.105926 31056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:28.106148 31056 log.cc:824] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:28.106799 31056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f: No bootstrap required, opened a new log
I20260925 14:54:28.108345 31056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458139ef94d648ad8199f82098a14b2f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44929 } }
I20260925 14:54:28.108474 31056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:28.108495 31056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458139ef94d648ad8199f82098a14b2f, State: Initialized, Role: FOLLOWER
I20260925 14:54:28.108608 31056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458139ef94d648ad8199f82098a14b2f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44929 } }
I20260925 14:54:28.108659 31056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:28.108678 31056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:28.108704 31056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:28.109210 31056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458139ef94d648ad8199f82098a14b2f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44929 } }
I20260925 14:54:28.109290 31056 leader_election.cc:302] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458139ef94d648ad8199f82098a14b2f; no voters:
I20260925 14:54:28.109457 31056 leader_election.cc:288] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:28.109755 31056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:28.109772 31061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:28.109910 31061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [term 1 LEADER]: Becoming Leader. State: Replica: 458139ef94d648ad8199f82098a14b2f, State: Running, Role: LEADER
I20260925 14:54:28.109999 31061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458139ef94d648ad8199f82098a14b2f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44929 } }
I20260925 14:54:28.110359 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458139ef94d648ad8199f82098a14b2f. Latest consensus state: current_term: 1 leader_uuid: "458139ef94d648ad8199f82098a14b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458139ef94d648ad8199f82098a14b2f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44929 } } }
I20260925 14:54:28.110484 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:28.110352 31062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458139ef94d648ad8199f82098a14b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458139ef94d648ad8199f82098a14b2f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44929 } } }
I20260925 14:54:28.110646 31062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:28.110697 31070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:28.111145 31070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:28.112721 31070 catalog_manager.cc:1347] Generated new cluster ID: 82191efa6435491bba413ed0f559cc99
I20260925 14:54:28.112775 31070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:28.130013 31070 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:28.130185 31070 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:28.138859 31070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458139ef94d648ad8199f82098a14b2f: Generated new TSK 0
I20260925 14:54:28.139045 31070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:28.145216 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:44929
--builtin_ntp_servers=127.20.199.20:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:54:28.226536 31080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:28.226783 31080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:28.226819 31080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:28.227982 31080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:28.228053 31080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:28.228097 31080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:28.229532 31080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:28.229782 31080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:28.230079 31080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:28.235030 31085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:28.235127 31086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:28.235037 31088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:28.235567 31080 server_base.cc:1034] running on GCE node
I20260925 14:54:28.235785 31080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:28.236027 31080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:28.237208 31080 hybrid_clock.cc:648] HybridClock initialized: now 1790348068237187 us; error 34 us; skew 500 ppm
I20260925 14:54:28.238585 31080 webserver.cc:460] Webserver started at http://127.20.199.1:36095/ using document root <none> and password file <none>
I20260925 14:54:28.238827 31080 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:28.238894 31080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:28.239040 31080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:28.240154 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "990e64d1ea4b4814a2043f5b592ce8a0"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "1515e9137e17828db413089dc5f614cd"
server_key_iv: "b90ef5da92338cc4d5fc75f89a942ee9"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.240526 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "990e64d1ea4b4814a2043f5b592ce8a0"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "1515e9137e17828db413089dc5f614cd"
server_key_iv: "b90ef5da92338cc4d5fc75f89a942ee9"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.241932 31080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:54:28.242676 31094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.242847 31080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:28.242944 31080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "990e64d1ea4b4814a2043f5b592ce8a0"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "1515e9137e17828db413089dc5f614cd"
server_key_iv: "b90ef5da92338cc4d5fc75f89a942ee9"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.243037 31080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:28.246351 31080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:28.246608 31080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:28.246753 31080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:28.246954 31080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:28.247257 31080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:28.247313 31080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.247350 31080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:28.247375 31080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.252983 31080 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:42787
I20260925 14:54:28.253063 31207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:42787 every 8 connection(s)
I20260925 14:54:28.253386 31080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:28.260133 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31080
I20260925 14:54:28.260255 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:28.260459 21276 external_mini_cluster.cc:1428] Setting key 3f3fc339543da8a79e3922b7efdc3ee7
I20260925 14:54:28.260521 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.254162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59067 (local address 127.20.199.62:44929)
0925 14:54:28.254244 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:28.254246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:28.254471 (+ 225us) server_negotiation.cc:407] Connection header received
0925 14:54:28.254945 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:28.254946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:28.254952 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:28.255016 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:28.260187 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:28.260189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:28.260189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:28.260194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:28.260238 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:28.260244 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:28.260313 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:28.260431 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:54:28.260448 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:54:28.260792 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.254124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:44929 (local address 127.20.199.1:59067)
0925 14:54:28.254334 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:28.254366 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:28.254825 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:28.255076 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:28.255079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:28.255087 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:28.259994 (+ 4907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:28.259994 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:28.260037 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:28.260079 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:28.260328 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:28.260336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:28.260339 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:28.260443 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:54:28.260541 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260925 14:54:28.261253 31208 heartbeater.cc:347] Connected to a master server at 127.20.199.62:44929
I20260925 14:54:28.261358 31208 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:28.261528 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:44929
--builtin_ntp_servers=127.20.199.20:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 14:54:28.261540 31208 heartbeater.cc:511] Master 127.20.199.62:44929 requested a full tablet report, sending...
I20260925 14:54:28.262066 31020 ts_manager.cc:194] Registered new tserver with Master: 990e64d1ea4b4814a2043f5b592ce8a0 (127.20.199.1:42787)
W20260925 14:54:28.341153 31211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:28.341341 31211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:28.341362 31211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:28.342527 31211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:28.342567 31211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:28.342592 31211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:28.343886 31211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:28.344133 31211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:28.344431 31211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:28.349334 31219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:28.349454 31211 server_base.cc:1034] running on GCE node
W20260925 14:54:28.349308 31216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:28.349380 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:28.349782 31211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:28.349980 31211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:28.351168 31211 hybrid_clock.cc:648] HybridClock initialized: now 1790348068351147 us; error 36 us; skew 500 ppm
I20260925 14:54:28.352377 31211 webserver.cc:460] Webserver started at http://127.20.199.2:45177/ using document root <none> and password file <none>
I20260925 14:54:28.352648 31211 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:28.352696 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:28.352861 31211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:28.353961 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "5c72228dbb6d461f888bcd2923899732"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "fc35ee631d95143060d16599c8109e3a"
server_key_iv: "9c1d68a243a866cc605096389f2fb613"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.354329 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "5c72228dbb6d461f888bcd2923899732"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "fc35ee631d95143060d16599c8109e3a"
server_key_iv: "9c1d68a243a866cc605096389f2fb613"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.355597 31211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:54:28.356307 31225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.356479 31211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:28.356592 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "5c72228dbb6d461f888bcd2923899732"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "fc35ee631d95143060d16599c8109e3a"
server_key_iv: "9c1d68a243a866cc605096389f2fb613"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.356666 31211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:28.359959 31211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:28.360203 31211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:28.360339 31211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:28.360538 31211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:28.360828 31211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:28.360860 31211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.360911 31211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:28.360939 31211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.366681 31211 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:38661
I20260925 14:54:28.366752 31338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:38661 every 8 connection(s)
I20260925 14:54:28.367062 31211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:28.369911 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.367766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:56517 (local address 127.20.199.62:44929)
0925 14:54:28.367843 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:28.367846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:28.368032 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:54:28.368454 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:28.368456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:28.368460 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:28.368557 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:28.369712 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:28.369713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:28.369714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:28.369718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:28.369732 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:28.369736 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:28.369800 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:28.369847 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:54:28.369859 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:54:28.370086 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.367731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:44929 (local address 127.20.199.2:56517)
0925 14:54:28.367922 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:28.367953 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:28.368374 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:28.368610 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:28.368613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:28.368620 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:28.369554 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:28.369555 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:28.369582 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:28.369609 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:28.369799 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:28.369802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:28.369804 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:54:28.369853 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:54:28.369896 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260925 14:54:28.370687 31339 heartbeater.cc:347] Connected to a master server at 127.20.199.62:44929
I20260925 14:54:28.370779 31339 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:28.370914 31339 heartbeater.cc:511] Master 127.20.199.62:44929 requested a full tablet report, sending...
I20260925 14:54:28.371284 31020 ts_manager.cc:194] Registered new tserver with Master: 5c72228dbb6d461f888bcd2923899732 (127.20.199.2:38661)
I20260925 14:54:28.376268 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31211
I20260925 14:54:28.376358 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:28.376574 21276 external_mini_cluster.cc:1428] Setting key d61fc44937bf3e1a4afb4fb3e23ab410
I20260925 14:54:28.377602 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:44929
--builtin_ntp_servers=127.20.199.20:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:54:28.471206 31342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:28.471405 31342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:28.471426 31342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:28.472570 31342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:28.472609 31342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:28.472635 31342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:28.474035 31342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:28.474269 31342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:28.474561 31342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:28.479565 31348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:28.479573 31350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:28.479574 31347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:28.479590 31342 server_base.cc:1034] running on GCE node
I20260925 14:54:28.479990 31342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:28.480266 31342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:28.481426 31342 hybrid_clock.cc:648] HybridClock initialized: now 1790348068481391 us; error 26 us; skew 500 ppm
I20260925 14:54:28.482895 31342 webserver.cc:460] Webserver started at http://127.20.199.3:41713/ using document root <none> and password file <none>
I20260925 14:54:28.483165 31342 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:28.483258 31342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:28.483392 31342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:28.484476 31342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "d2e7aed6db7f4060bddabc1977c9b4ec"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "dcdc6208ef8b4c995c2822e6e2e74f9c"
server_key_iv: "c7ed7f04579c7350b10773268bbb2f78"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.484779 31342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e7aed6db7f4060bddabc1977c9b4ec"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "dcdc6208ef8b4c995c2822e6e2e74f9c"
server_key_iv: "c7ed7f04579c7350b10773268bbb2f78"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.486083 31342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:28.486778 31356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.486949 31342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:28.487032 31342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "d2e7aed6db7f4060bddabc1977c9b4ec"
format_stamp: "Formatted at 2026-09-25 14:54:28 on dist-test-slave-8qt6"
server_key: "dcdc6208ef8b4c995c2822e6e2e74f9c"
server_key_iv: "c7ed7f04579c7350b10773268bbb2f78"
server_key_version: "encryptionkey@0"
I20260925 14:54:28.487099 31342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:28.490358 31342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:28.490609 31342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:28.490739 31342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:28.490952 31342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:28.491245 31342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:28.491297 31342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.491334 31342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:28.491360 31342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:28.497129 31342 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:39475
I20260925 14:54:28.497198 31469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:39475 every 8 connection(s)
I20260925 14:54:28.497581 31342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:28.500438 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.498281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:47915 (local address 127.20.199.62:44929)
0925 14:54:28.498326 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:28.498328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:28.498528 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:54:28.499033 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:28.499034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:28.499040 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:28.499094 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:28.500205 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:28.500206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:28.500207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:28.500211 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:28.500227 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:28.500231 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:28.500284 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:28.500356 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:54:28.500370 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:28.500638 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:28.498246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:44929 (local address 127.20.199.3:47915)
0925 14:54:28.498396 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:28.498435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:28.498916 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:28.499171 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:28.499175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:28.499184 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:28.500055 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:28.500055 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:28.500081 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:28.500108 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:28.500281 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:28.500285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:28.500289 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:28.500366 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:54:28.500425 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260925 14:54:28.501137 31470 heartbeater.cc:347] Connected to a master server at 127.20.199.62:44929
I20260925 14:54:28.501236 31470 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:28.501390 31470 heartbeater.cc:511] Master 127.20.199.62:44929 requested a full tablet report, sending...
I20260925 14:54:28.501792 31020 ts_manager.cc:194] Registered new tserver with Master: d2e7aed6db7f4060bddabc1977c9b4ec (127.20.199.3:39475)
I20260925 14:54:28.502213 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31342
I20260925 14:54:28.502310 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:28.502514 21276 external_mini_cluster.cc:1428] Setting key f6f64822c5a166b3760208ccc8cd65b6
I20260925 14:54:28.503332 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:54:28.505182 31059 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:28.504721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.20.199.62:44929)
0925 14:54:28.504761 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:28.504763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:28.504885 (+ 122us) server_negotiation.cc:407] Connection header received
0925 14:54:28.504933 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:28.504934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:28.504938 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:28.504999 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:28.505124 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59018: BlockingRecv error: recv got EOF from 127.0.0.1:59018 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:28.505256 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31080
I20260925 14:54:28.510489 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31211
I20260925 14:54:28.515126 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31342
I20260925 14:54:28.519379 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 30989
2026-09-25T14:54:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:28 dist-test-slave-8qt6 krb5kdc[31480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:28 dist-test-slave-8qt6 krb5kdc[31480](info): set up 2 sockets
Sep 25 14:54:28 dist-test-slave-8qt6 krb5kdc[31480](info): commencing operation
krb5kdc: starting...
W20260925 14:54:29.562314 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:54:29 dist-test-slave-8qt6 krb5kdc[31480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:54:29.678428 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38361
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:38361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:29.756297 31493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:29.756490 31493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:29.756512 31493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:29.757668 31493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:29.757706 31493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:29.757720 31493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:29.757732 31493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:29.757745 31493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:29.759524 31493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:38361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:38361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:29.759800 31493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:29.760037 31493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:29.765045 31501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:29.765049 31499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:29.765064 31493 server_base.cc:1034] running on GCE node
W20260925 14:54:29.765043 31498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:29.765538 31493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:29.765844 31493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:29.766981 31493 hybrid_clock.cc:648] HybridClock initialized: now 1790348069766961 us; error 39 us; skew 500 ppm
Sep 25 14:54:29 dist-test-slave-8qt6 krb5kdc[31480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:29.769465 31493 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:54:29.769898 31493 webserver.cc:460] Webserver started at http://127.20.199.62:34301/ using document root <none> and password file <none>
I20260925 14:54:29.770083 31493 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:29.770123 31493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:29.770208 31493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:29.771142 31493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "61b25152842949ada1afdf67cfc8c8e8"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-8qt6"
server_key: "0a6a958f5a096ac2c5db302aa182a430"
server_key_iv: "4ea87ab48747c5ce41f33f4b0c643987"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.771427 31493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "61b25152842949ada1afdf67cfc8c8e8"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-8qt6"
server_key: "0a6a958f5a096ac2c5db302aa182a430"
server_key_iv: "4ea87ab48747c5ce41f33f4b0c643987"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.772607 31493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:29.773288 31508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:29.773475 31493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:29.773572 31493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "61b25152842949ada1afdf67cfc8c8e8"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-8qt6"
server_key: "0a6a958f5a096ac2c5db302aa182a430"
server_key_iv: "4ea87ab48747c5ce41f33f4b0c643987"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.773667 31493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:29.787691 31493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:29.789575 31493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:29.789738 31493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:29.793720 31493 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:38361
I20260925 14:54:29.793769 31560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:38361 every 8 connection(s)
I20260925 14:54:29.794114 31493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:29.795065 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:29.797276 31561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8: Bootstrap starting.
I20260925 14:54:29.797863 31561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:29.798151 31561 log.cc:824] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:29.798815 31561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8: No bootstrap required, opened a new log
I20260925 14:54:29.800361 31561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b25152842949ada1afdf67cfc8c8e8" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38361 } }
I20260925 14:54:29.800495 31561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:29.800585 31561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b25152842949ada1afdf67cfc8c8e8, State: Initialized, Role: FOLLOWER
I20260925 14:54:29.800729 31561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b25152842949ada1afdf67cfc8c8e8" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38361 } }
I20260925 14:54:29.800787 31561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:29.800807 31561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:29.800870 31561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:29.801760 31561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b25152842949ada1afdf67cfc8c8e8" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38361 } }
I20260925 14:54:29.801872 31561 leader_election.cc:302] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61b25152842949ada1afdf67cfc8c8e8; no voters:
I20260925 14:54:29.802047 31561 leader_election.cc:288] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:29.802110 31564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:29.802277 31564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [term 1 LEADER]: Becoming Leader. State: Replica: 61b25152842949ada1afdf67cfc8c8e8, State: Running, Role: LEADER
I20260925 14:54:29.802302 31561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:29.802369 31564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b25152842949ada1afdf67cfc8c8e8" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38361 } }
I20260925 14:54:29.802699 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31493
I20260925 14:54:29.802821 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:54:29.803020 21276 external_mini_cluster.cc:1428] Setting key 2040bfa5702340e8eff11a008ba88e1a
I20260925 14:54:29.803489 31565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61b25152842949ada1afdf67cfc8c8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b25152842949ada1afdf67cfc8c8e8" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38361 } } }
I20260925 14:54:29.803602 31565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:29.803714 31566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b25152842949ada1afdf67cfc8c8e8. Latest consensus state: current_term: 1 leader_uuid: "61b25152842949ada1afdf67cfc8c8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b25152842949ada1afdf67cfc8c8e8" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 38361 } } }
I20260925 14:54:29.804033 31566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:29.803966 31582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:29.804529 31582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:29.806097 31582 catalog_manager.cc:1347] Generated new cluster ID: c56913084cf540ffbc235796fb4ac1b4
I20260925 14:54:29.806150 31582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:29 dist-test-slave-8qt6 krb5kdc[31480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:29.812930 31582 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:29.813272 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:29.803527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.20.199.62:38361)
0925 14:54:29.803696 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:29.803706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:29.804046 (+ 340us) server_negotiation.cc:407] Connection header received
0925 14:54:29.804167 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:29.804174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:29.804229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:29.804364 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:29.805016 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.806342 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.807085 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.807506 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.807628 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:29.809914 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:29.809917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:29.809918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:29.809935 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:29.811572 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.811954 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.811958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.811958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.812054 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.812272 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.812273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.812274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.812497 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:29.812873 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:29.812953 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:54:29.813120 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260925 14:54:29.813755 31582 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:29.827476 31582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8: Generated new TSK 0
I20260925 14:54:29.827654 31582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:54:29.856468 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38361
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:29.939006 31587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:29.939311 31587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:29.939352 31587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:29.941037 31587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:29.941082 31587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:29.941110 31587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:54:29.942754 31587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:29.943003 31587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:29.943255 31587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:29.948271 31592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:29.948284 31595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:29.948287 31593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:29.948304 31587 server_base.cc:1034] running on GCE node
I20260925 14:54:29.948709 31587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:29.948975 31587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:29.950114 31587 hybrid_clock.cc:648] HybridClock initialized: now 1790348069950107 us; error 29 us; skew 500 ppm
Sep 25 14:54:29 dist-test-slave-8qt6 krb5kdc[31480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:29.952657 31587 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:54:29.953109 31587 webserver.cc:460] Webserver started at http://127.20.199.1:36935/ using document root <none> and password file <none>
I20260925 14:54:29.953354 31587 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:29.953451 31587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:29.953601 31587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:29.954591 31587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "574e1d2d2f3c472bab43bfbe947c79f2"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-8qt6"
server_key: "e243122d8fd76e60872c725ea85663f2"
server_key_iv: "2ee0a470f86db1871dcb07cdebe18ec5"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.954958 31587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "574e1d2d2f3c472bab43bfbe947c79f2"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-8qt6"
server_key: "e243122d8fd76e60872c725ea85663f2"
server_key_iv: "2ee0a470f86db1871dcb07cdebe18ec5"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.956244 31587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:29.956960 31602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:29.957154 31587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:29.957278 31587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "574e1d2d2f3c472bab43bfbe947c79f2"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-8qt6"
server_key: "e243122d8fd76e60872c725ea85663f2"
server_key_iv: "2ee0a470f86db1871dcb07cdebe18ec5"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.957358 31587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:29.968904 31587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:29.971166 31587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:29.971320 31587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:29.971530 31587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:29.971828 31587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:29.971880 31587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:29.971916 31587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:29.971942 31587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:29.978044 31587 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:39589
I20260925 14:54:29.978097 31715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:39589 every 8 connection(s)
I20260925 14:54:29.978394 31587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:29.981292 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31587
I20260925 14:54:29.981494 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:29.981750 21276 external_mini_cluster.cc:1428] Setting key c8693807a5fd444aad065874827c49d8
Sep 25 14:54:29 dist-test-slave-8qt6 krb5kdc[31480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:29.988492 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:29.979053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:53009 (local address 127.20.199.62:38361)
0925 14:54:29.979132 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:29.979135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:29.979292 (+ 157us) server_negotiation.cc:407] Connection header received
0925 14:54:29.979805 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:29.979806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:29.979817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:29.979869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:29.980786 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.981276 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.982744 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.982935 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.982991 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:29.985728 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:29.985730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:29.985730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:29.985738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:29.987095 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.987497 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.987498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.987499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.987575 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.987829 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.987830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.987830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.987946 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:29.988024 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:29.988307 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 14:54:29.988389 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:54:29.988694 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:29.979016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38361 (local address 127.20.199.1:53009)
0925 14:54:29.979174 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:29.979209 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:29.979719 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:29.979924 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:29.979927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:29.980519 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:29.980709 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:29.980714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.981424 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.981425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:29.982662 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:29.982666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.983019 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.983020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:29.983178 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:29.983687 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:29.983697 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:29.985634 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:29.987194 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:29.987201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:29.987203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:29.987418 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:29.987644 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:29.987645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:29.987646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:29.987757 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:29.988016 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:29.988022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:29.988160 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:54:29.988284 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:54:29.988444 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260925 14:54:29.989131 31716 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38361
I20260925 14:54:29.989276 31716 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:29.989508 31716 heartbeater.cc:511] Master 127.20.199.62:38361 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
I20260925 14:54:29.990041 31525 ts_manager.cc:194] Registered new tserver with Master: 574e1d2d2f3c472bab43bfbe947c79f2 (127.20.199.1:39589)
I20260925 14:54:29.990700 31525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:53009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:54:30.008198 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38361
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:30.087292 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:30.087497 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:30.087518 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:30.088585 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:30.088620 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:30.088645 31721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:54:30.090026 31721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:30.090205 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:30.090366 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:30.095232 31726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:30.095288 31727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:30.095306 31729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:30.095498 31721 server_base.cc:1034] running on GCE node
I20260925 14:54:30.095710 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:30.095937 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:30.097088 31721 hybrid_clock.cc:648] HybridClock initialized: now 1790348070097070 us; error 37 us; skew 500 ppm
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[31480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348070, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:30.099509 31721 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:54:30.099941 31721 webserver.cc:460] Webserver started at http://127.20.199.2:36013/ using document root <none> and password file <none>
I20260925 14:54:30.100168 31721 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:30.100250 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:30.100400 31721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:30.101473 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "46f91d53b9c249ada3bde85638d3cdf2"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-8qt6"
server_key: "3cf7fea03195c7c16d411ccd9acaa4e2"
server_key_iv: "4a8853d53f226bbfc93c6cc17b8a1ec6"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.101848 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "46f91d53b9c249ada3bde85638d3cdf2"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-8qt6"
server_key: "3cf7fea03195c7c16d411ccd9acaa4e2"
server_key_iv: "4a8853d53f226bbfc93c6cc17b8a1ec6"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.103351 31721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:30.104033 31736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.104210 31721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:30.104316 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "46f91d53b9c249ada3bde85638d3cdf2"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-8qt6"
server_key: "3cf7fea03195c7c16d411ccd9acaa4e2"
server_key_iv: "4a8853d53f226bbfc93c6cc17b8a1ec6"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.104398 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:30.132966 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:30.135346 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:30.135511 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:30.135756 31721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:30.136068 31721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:30.136123 31721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.136169 31721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:30.136201 31721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.142021 31721 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:36427
I20260925 14:54:30.142097 31849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:36427 every 8 connection(s)
I20260925 14:54:30.142405 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:30.143023 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31721
I20260925 14:54:30.143123 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:30.143332 21276 external_mini_cluster.cc:1428] Setting key 16ddd48a1bbfedeb476b36e7b0e08ec8
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[31480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348070, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:54:30.152299 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.143180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:43027 (local address 127.20.199.62:38361)
0925 14:54:30.143226 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.143228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.143444 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:54:30.144120 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.144121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.144134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.144204 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:30.145082 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.145739 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.146735 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.146941 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.147031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.149333 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:30.149334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:30.149335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:30.149343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:30.150888 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:30.151299 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:30.151301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:30.151301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:30.151363 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:30.151652 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:30.151653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:30.151653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:30.151768 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:30.151839 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.152123 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.152209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:30.152468 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.143163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38361 (local address 127.20.199.2:43027)
0925 14:54:30.143312 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.143346 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.144018 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.144259 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.144262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.144810 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:30.144992 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.144996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.145839 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.145840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.146631 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.146633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.147039 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.147040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.147167 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.147662 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:30.147671 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:30.149236 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:30.150985 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:30.150991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:30.150993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:30.151209 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:30.151431 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:30.151431 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:30.151432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:30.151541 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:30.151844 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:30.151851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:30.151989 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.152088 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.152238 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 14:54:30.152879 31850 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38361
I20260925 14:54:30.152983 31850 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:30.153152 31850 heartbeater.cc:511] Master 127.20.199.62:38361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
I20260925 14:54:30.153558 31525 ts_manager.cc:194] Registered new tserver with Master: 46f91d53b9c249ada3bde85638d3cdf2 (127.20.199.2:36427)
I20260925 14:54:30.153920 31525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:43027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:54:30.168291 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38361
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:30.246824 31855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:30.247017 31855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:30.247040 31855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:30.248078 31855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:30.248112 31855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:30.248137 31855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:54:30.249521 31855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:38361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:54:30.249701 31855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:30.249859 31855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:30.254717 31861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:30.254769 31855 server_base.cc:1034] running on GCE node
W20260925 14:54:30.254722 31860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:30.254716 31863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:30.255108 31855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:30.255383 31855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:30.256533 31855 hybrid_clock.cc:648] HybridClock initialized: now 1790348070256512 us; error 34 us; skew 500 ppm
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[31480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348070, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:30.259037 31855 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:54:30.259457 31855 webserver.cc:460] Webserver started at http://127.20.199.3:36063/ using document root <none> and password file <none>
I20260925 14:54:30.259720 31855 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:30.259764 31855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:30.259902 31855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:30.261060 31855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "cea0564528d3414c9a0acdc92478eff1"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-8qt6"
server_key: "031d5b60a39fad4a2294342f2a40ad8f"
server_key_iv: "d2bb8716643db65d8e381350c5f9b6b1"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.261445 31855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "cea0564528d3414c9a0acdc92478eff1"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-8qt6"
server_key: "031d5b60a39fad4a2294342f2a40ad8f"
server_key_iv: "d2bb8716643db65d8e381350c5f9b6b1"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.262826 31855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:30.263523 31870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.263692 31855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:30.263784 31855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "cea0564528d3414c9a0acdc92478eff1"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-8qt6"
server_key: "031d5b60a39fad4a2294342f2a40ad8f"
server_key_iv: "d2bb8716643db65d8e381350c5f9b6b1"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.263859 31855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:30.283799 31855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:30.286129 31855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:30.286281 31855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:30.286530 31855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:30.286819 31855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:30.286871 31855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.286911 31855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:30.286939 31855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.292795 31855 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:33731
I20260925 14:54:30.292865 31983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:33731 every 8 connection(s)
I20260925 14:54:30.293195 31855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[31480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348070, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:54:30.301640 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.293966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:57177 (local address 127.20.199.62:38361)
0925 14:54:30.294039 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.294041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.294208 (+ 167us) server_negotiation.cc:407] Connection header received
0925 14:54:30.294643 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.294644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.294654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.294721 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:30.295576 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.296055 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.296974 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.297155 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.297206 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.299140 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:30.299141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:30.299142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:30.299148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:30.300500 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:30.300839 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:30.300839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:30.300840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:30.300886 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:30.301083 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:30.301083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:30.301084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:30.301178 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:30.301227 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.301491 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.301579 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:54:30.301800 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.293909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:38361 (local address 127.20.199.3:57177)
0925 14:54:30.294091 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.294124 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.294568 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.294773 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.294776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.295322 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:30.295505 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.295509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.296127 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.296128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.296889 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.296891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.297224 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.297225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.297318 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.297725 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:30.297733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:30.299064 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:30.300569 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:30.300573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:30.300575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:30.300776 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:30.300941 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:30.300942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:30.300942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:30.301030 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:30.301230 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:30.301233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:30.301335 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.301476 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.301600 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260925 14:54:30.302294 31984 heartbeater.cc:347] Connected to a master server at 127.20.199.62:38361
I20260925 14:54:30.302381 31984 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:30.302505 31984 heartbeater.cc:511] Master 127.20.199.62:38361 requested a full tablet report, sending...
I20260925 14:54:30.302848 31525 ts_manager.cc:194] Registered new tserver with Master: cea0564528d3414c9a0acdc92478eff1 (127.20.199.3:33731)
I20260925 14:54:30.302933 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 31855
I20260925 14:54:30.303059 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:30.303270 21276 external_mini_cluster.cc:1428] Setting key 2937714a89b5876008be1e05006a87a5
I20260925 14:54:30.303292 31525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:57177
I20260925 14:54:30.304227 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:30.311167 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.305502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.20.199.62:38361)
0925 14:54:30.305579 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.305581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.305704 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:54:30.305779 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.305780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.305790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.305844 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:30.306392 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.306872 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.307594 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.307762 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.307811 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.308860 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:30.308861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:30.308861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:30.308867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:30.310316 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:30.310589 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:30.310590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:30.310590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:30.310630 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:30.310766 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:30.310766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:30.310767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:30.310873 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:30.310941 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.311032 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.311084 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:54:30.312532 31525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:30.314139 31525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:30.320580 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.316457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.20.199.2:36427)
0925 14:54:30.317048 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.317053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.317062 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:30.317100 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.317102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.317117 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.317204 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:30.317421 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.318452 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.319704 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.320202 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.320290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.320347 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.320465 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.320506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":80,"threads_started":1}
I20260925 14:54:30.320659 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.316543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33731 (local address 127.0.0.1:52158)
0925 14:54:30.316890 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.316900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.316955 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.317222 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.317222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.317224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:30.317292 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.317294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.318174 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.318175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.319522 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.319530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.320440 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.320441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.320516 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.320519 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.320563 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.320600 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":53,"threads_started":1}
I20260925 14:54:30.320822 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.316560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.20.199.1:39589)
0925 14:54:30.316750 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.316756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.316765 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:30.316804 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.316806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.316828 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.316919 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:30.317098 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.318260 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.319531 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.320285 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.320614 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.320676 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.320701 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.320748 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260925 14:54:30.320937 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.316721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.20.199.3:33731)
0925 14:54:30.316883 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.316888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.316942 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:54:30.317060 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.317062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.317077 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.317157 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:30.317364 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.318071 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.319620 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.320293 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.320359 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.320413 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.320767 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.320810 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260925 14:54:30.321041 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.316364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:39589 (local address 127.0.0.1:57010)
0925 14:54:30.316511 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.316548 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.316605 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.316923 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.316926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.316929 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:30.317009 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.317011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.318375 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.318376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.319431 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.319433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.320391 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.320391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.320468 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.320472 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.320504 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.320554 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":85,"thread_start_us":53,"threads_started":1}
I20260925 14:54:30.320574 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.316364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:36427 (local address 127.0.0.1:52298)
0925 14:54:30.316635 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.316646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.316701 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.317281 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.317282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.317285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:30.317323 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.317325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.318553 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.318554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.319631 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.319632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.320289 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.320290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.320367 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.320379 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.320450 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.320501 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":3,"thread_start_us":51,"threads_started":1}
I20260925 14:54:30.321671 31918 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5a25a8460b479eb01904b852fc748d (DEFAULT_TABLE table=test-table [id=ab9def375ac14100b00319eff8d9d788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:30.321763 31650 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5a25a8460b479eb01904b852fc748d (DEFAULT_TABLE table=test-table [id=ab9def375ac14100b00319eff8d9d788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:30.322005 31918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5a25a8460b479eb01904b852fc748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:30.321957 31784 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5a25a8460b479eb01904b852fc748d (DEFAULT_TABLE table=test-table [id=ab9def375ac14100b00319eff8d9d788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:30.322124 31650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5a25a8460b479eb01904b852fc748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:30.322232 31784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5a25a8460b479eb01904b852fc748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:30.324654 31998 tablet_bootstrap.cc:492] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: Bootstrap starting.
I20260925 14:54:30.325232 32000 tablet_bootstrap.cc:492] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Bootstrap starting.
I20260925 14:54:30.325356 31998 tablet_bootstrap.cc:654] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:30.325557 31999 tablet_bootstrap.cc:492] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Bootstrap starting.
I20260925 14:54:30.325932 32000 tablet_bootstrap.cc:654] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:30.325994 31998 log.cc:824] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:30.326131 31999 tablet_bootstrap.cc:654] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:30.326489 32000 log.cc:824] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:30.326671 31999 log.cc:824] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:30.326809 31998 tablet_bootstrap.cc:492] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: No bootstrap required, opened a new log
I20260925 14:54:30.326871 31998 ts_tablet_manager.cc:1397] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:30.327203 32000 tablet_bootstrap.cc:492] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: No bootstrap required, opened a new log
I20260925 14:54:30.327256 32000 ts_tablet_manager.cc:1397] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:30.327334 31999 tablet_bootstrap.cc:492] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: No bootstrap required, opened a new log
I20260925 14:54:30.327376 31999 ts_tablet_manager.cc:1397] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:30.328815 31998 raft_consensus.cc:348] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.328884 31999 raft_consensus.cc:348] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.328969 31998 raft_consensus.cc:374] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:30.328995 31999 raft_consensus.cc:374] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:30.329005 31998 raft_consensus.cc:729] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574e1d2d2f3c472bab43bfbe947c79f2, State: Initialized, Role: FOLLOWER
I20260925 14:54:30.329016 31999 raft_consensus.cc:729] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea0564528d3414c9a0acdc92478eff1, State: Initialized, Role: FOLLOWER
I20260925 14:54:30.329123 31998 consensus_queue.cc:260] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.329159 31999 consensus_queue.cc:260] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.329087 32000 raft_consensus.cc:348] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.329241 32000 raft_consensus.cc:374] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:30.329283 32000 raft_consensus.cc:729] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f91d53b9c249ada3bde85638d3cdf2, State: Initialized, Role: FOLLOWER
I20260925 14:54:30.329440 31998 ts_tablet_manager.cc:1428] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:30.329455 31999 ts_tablet_manager.cc:1428] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:30.329435 32000 consensus_queue.cc:260] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.329484 31716 heartbeater.cc:503] Master 127.20.199.62:38361 was elected leader, sending a full tablet report...
I20260925 14:54:30.329571 31984 heartbeater.cc:503] Master 127.20.199.62:38361 was elected leader, sending a full tablet report...
I20260925 14:54:30.329692 32000 ts_tablet_manager.cc:1428] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:54:30.329789 31850 heartbeater.cc:503] Master 127.20.199.62:38361 was elected leader, sending a full tablet report...
W20260925 14:54:30.392966 31851 tablet.cc:2369] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:30.479135 31717 tablet.cc:2369] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:30.543768 31985 tablet.cc:2369] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:30.629914 32005 raft_consensus.cc:482] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:30.630044 32005 raft_consensus.cc:504] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.630519 32005 leader_election.cc:288] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 574e1d2d2f3c472bab43bfbe947c79f2 (127.20.199.1:39589), 46f91d53b9c249ada3bde85638d3cdf2 (127.20.199.2:36427)
I20260925 14:54:30.634469 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.630681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:39589 (local address 127.20.199.3:33599)
0925 14:54:30.630768 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.630779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.630826 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.631022 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.631023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.631025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:30.631115 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.631117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.632030 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.632030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.633372 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.633375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.634169 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.634170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.634249 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.634253 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.634314 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.634368 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":126}
I20260925 14:54:30.634475 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.630794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:33299 (local address 127.20.199.2:36427)
0925 14:54:30.630848 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.630850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.630918 (+ 68us) server_negotiation.cc:407] Connection header received
0925 14:54:30.630950 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.630950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.630959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.631026 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:30.631202 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.631948 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.633510 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.634090 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.634154 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.634162 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.634357 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.634405 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:54:30.634507 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.630787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:33599 (local address 127.20.199.1:39589)
0925 14:54:30.630835 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.630837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.630853 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:54:30.630888 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.630889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.630901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.630983 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:30.631202 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.631947 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.633514 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.634069 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.634153 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.634171 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.634360 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.634405 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:54:30.634475 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.630696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:36427 (local address 127.20.199.3:33299)
0925 14:54:30.630845 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.630855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:30.630911 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:30.631028 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:30.631028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:30.631029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:30.631115 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.631117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.632030 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.632031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.633372 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:30.633375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.634168 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.634168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:30.634249 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.634254 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:30.634314 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:54:30.634368 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":92,"thread_start_us":93,"threads_started":1}
I20260925 14:54:30.634831 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5a25a8460b479eb01904b852fc748d" candidate_uuid: "cea0564528d3414c9a0acdc92478eff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" is_pre_election: true
I20260925 14:54:30.635020 31670 raft_consensus.cc:2454] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea0564528d3414c9a0acdc92478eff1 in term 0.
I20260925 14:54:30.635039 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5a25a8460b479eb01904b852fc748d" candidate_uuid: "cea0564528d3414c9a0acdc92478eff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f91d53b9c249ada3bde85638d3cdf2" is_pre_election: true
I20260925 14:54:30.635200 31804 raft_consensus.cc:2454] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea0564528d3414c9a0acdc92478eff1 in term 0.
I20260925 14:54:30.635286 31873 leader_election.cc:302] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574e1d2d2f3c472bab43bfbe947c79f2, cea0564528d3414c9a0acdc92478eff1; no voters:
I20260925 14:54:30.635455 32005 raft_consensus.cc:2781] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:30.635547 32005 raft_consensus.cc:482] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:30.635565 32005 raft_consensus.cc:3037] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:30.636318 32005 raft_consensus.cc:504] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.636462 32005 leader_election.cc:288] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [CANDIDATE]: Term 1 election: Requested vote from peers 574e1d2d2f3c472bab43bfbe947c79f2 (127.20.199.1:39589), 46f91d53b9c249ada3bde85638d3cdf2 (127.20.199.2:36427)
I20260925 14:54:30.636627 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5a25a8460b479eb01904b852fc748d" candidate_uuid: "cea0564528d3414c9a0acdc92478eff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f91d53b9c249ada3bde85638d3cdf2"
I20260925 14:54:30.636674 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5a25a8460b479eb01904b852fc748d" candidate_uuid: "cea0564528d3414c9a0acdc92478eff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574e1d2d2f3c472bab43bfbe947c79f2"
I20260925 14:54:30.636703 31804 raft_consensus.cc:3037] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:30.636776 31670 raft_consensus.cc:3037] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:30.637499 31804 raft_consensus.cc:2454] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea0564528d3414c9a0acdc92478eff1 in term 1.
I20260925 14:54:30.637542 31670 raft_consensus.cc:2454] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea0564528d3414c9a0acdc92478eff1 in term 1.
I20260925 14:54:30.637698 31873 leader_election.cc:302] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574e1d2d2f3c472bab43bfbe947c79f2, cea0564528d3414c9a0acdc92478eff1; no voters:
I20260925 14:54:30.637853 32005 raft_consensus.cc:2781] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:30.637990 32005 raft_consensus.cc:686] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 1 LEADER]: Becoming Leader. State: Replica: cea0564528d3414c9a0acdc92478eff1, State: Running, Role: LEADER
I20260925 14:54:30.638098 32005 consensus_queue.cc:237] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } }
I20260925 14:54:30.638942 31524 catalog_manager.cc:5495] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cea0564528d3414c9a0acdc92478eff1 (127.20.199.3). New cstate: current_term: 1 leader_uuid: "cea0564528d3414c9a0acdc92478eff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea0564528d3414c9a0acdc92478eff1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33731 } health_report { overall_health: HEALTHY } } }
I20260925 14:54:30.672042 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.669783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.20.199.3:33731)
0925 14:54:30.669846 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.669848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.669856 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:30.669906 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.669906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.669922 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.669974 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:30.670109 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.670563 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.671305 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.671477 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.671525 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.671699 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:30.671793 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:30.671851 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.671926 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.671942 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:30.673854 31670 raft_consensus.cc:1261] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 1 FOLLOWER]: Refusing update from remote peer cea0564528d3414c9a0acdc92478eff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:30.673900 31804 raft_consensus.cc:1261] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 1 FOLLOWER]: Refusing update from remote peer cea0564528d3414c9a0acdc92478eff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:30.674146 32005 consensus_queue.cc:1035] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "574e1d2d2f3c472bab43bfbe947c79f2" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:30.674300 32008 consensus_queue.cc:1035] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f91d53b9c249ada3bde85638d3cdf2" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:30.676474 32011 mvcc.cc:205] Tried to move back new op lower bound from 7333265697477353472 to 7333265697334194176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265697477353472}]
I20260925 14:54:30.676782 32012 mvcc.cc:205] Tried to move back new op lower bound from 7333265697477353472 to 7333265697334194176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265697477353472}]
I20260925 14:54:30.677201 32013 mvcc.cc:205] Tried to move back new op lower bound from 7333265697477353472 to 7333265697334194176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265697477353472 or (T in {7333265697477353472})}]
I20260925 14:54:30.679046 31523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:30.679123 31523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:30.679893 31523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61b25152842949ada1afdf67cfc8c8e8: Sending DeleteTablet for 3 replicas of tablet 6e5a25a8460b479eb01904b852fc748d
I20260925 14:54:30.680225 31650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5a25a8460b479eb01904b852fc748d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:30 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:57010
I20260925 14:54:30.680240 31784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5a25a8460b479eb01904b852fc748d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:30 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:52298
I20260925 14:54:30.680266 31918 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5a25a8460b479eb01904b852fc748d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:30 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:52158
I20260925 14:54:30.680462 32021 tablet_replica.cc:330] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2: stopping tablet replica
I20260925 14:54:30.680580 32021 raft_consensus.cc:2229] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:30.680694 32021 raft_consensus.cc:2258] T 6e5a25a8460b479eb01904b852fc748d P 574e1d2d2f3c472bab43bfbe947c79f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:30.680826 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31587
I20260925 14:54:30.680859 32023 tablet_replica.cc:330] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: stopping tablet replica
I20260925 14:54:30.680938 32023 raft_consensus.cc:2229] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:30.681039 32023 raft_consensus.cc:2258] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:30.681576 32022 tablet_replica.cc:330] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: stopping tablet replica
I20260925 14:54:30.681663 32022 raft_consensus.cc:2229] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:30.681756 32022 raft_consensus.cc:2258] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:30.682367 32022 ts_tablet_manager.cc:1905] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:30.684139 32022 ts_tablet_manager.cc:1918] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:30.684233 32022 log.cc:1196] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/6e5a25a8460b479eb01904b852fc748d
I20260925 14:54:30.684468 32022 ts_tablet_manager.cc:1939] T 6e5a25a8460b479eb01904b852fc748d P 46f91d53b9c249ada3bde85638d3cdf2: Deleting consensus metadata
I20260925 14:54:30.684844 31509 catalog_manager.cc:4841] TS 46f91d53b9c249ada3bde85638d3cdf2 (127.20.199.2:36427): tablet 6e5a25a8460b479eb01904b852fc748d (table test-table [id=ab9def375ac14100b00319eff8d9d788]) successfully deleted
I20260925 14:54:30.685715 32023 ts_tablet_manager.cc:1905] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:30.686975 32023 ts_tablet_manager.cc:1918] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:30.687055 32023 log.cc:1196] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/6e5a25a8460b479eb01904b852fc748d
I20260925 14:54:30.687352 32023 ts_tablet_manager.cc:1939] T 6e5a25a8460b479eb01904b852fc748d P cea0564528d3414c9a0acdc92478eff1: Deleting consensus metadata
W20260925 14:54:30.687397 31511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:39589 (error 108)
I20260925 14:54:30.687634 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31721
I20260925 14:54:30.688537 31511 catalog_manager.cc:4841] TS cea0564528d3414c9a0acdc92478eff1 (127.20.199.3:33731): tablet 6e5a25a8460b479eb01904b852fc748d (table test-table [id=ab9def375ac14100b00319eff8d9d788]) successfully deleted
I20260925 14:54:30.693462 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31855
I20260925 14:54:30.693614 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.693460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:39589 (local address 127.0.0.1:57020)
0925 14:54:30.693536 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:30.693582 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:39589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 14:54:30.694083 31511 catalog_manager.cc:4584] TS 574e1d2d2f3c472bab43bfbe947c79f2 (127.20.199.1:39589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e5a25a8460b479eb01904b852fc748d: Network error: Client connection negotiation failed: client connection to 127.20.199.1:39589: connect: Connection refused (error 111)
I20260925 14:54:30.698280 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 31493
2026-09-25T14:54:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2179 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[32027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[32027](info): set up 2 sockets
Sep 25 14:54:30 dist-test-slave-8qt6 krb5kdc[32027](info): commencing operation
krb5kdc: starting...
W20260925 14:54:31.746433 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:54:31 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/rangeradmin_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/rangeradmin_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/rangerlookup_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/rangerlookup_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
I20260925 14:54:31.904018 21276 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyT1hpU/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/postgres -l logfile start
2026-09-25 14:54:34.352 UTC [32057] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:54:34.352 UTC [32057] LOG: listening on IPv4 address "127.20.199.20", port 56779
2026-09-25 14:54:34.355 UTC [32057] LOG: listening on Unix socket "/tmp/.s.PGSQL.56779"
2026-09-25 14:54:34.385 UTC [32060] LOG: database system was shut down at 2026-09-25 14:54:32 UTC
2026-09-25 14:54:34.389 UTC [32057] LOG: database system is ready to accept connections
I20260925 14:54:35.370350 21276 mini_postgres.cc:96] Postgres bound to 56779
2026-09-25 14:54:35.376 UTC [32068] FATAL: database "slave" does not exist
127.20.199.20:56779 - accepting connections
I20260925 14:54:35.376806 21276 mini_ranger.cc:161] Starting Ranger...
I20260925 14:54:35.390833 21276 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:54:35.860478 21276 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:54:35.860559 21276 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-admin
2026-09-25 14:54:36,128 [I] DB FLAVOR :POSTGRES
2026-09-25 14:54:36,128 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:54:36,128 [I] Checking connection..
2026-09-25 14:54:36,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:54:36,374 [I] Checking connection passed.
2026-09-25 14:54:36,375 [I] --------- Verifying version history table ---------
2026-09-25 14:54:36,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:54:36,620 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:54:36,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:54:36,844 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:54:36,844 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:54:36,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:54:37.080 UTC [32132] WARNING: there is no transaction in progress
2026-09-25 14:54:37,086 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:54:37,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:54:37,341 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:54:37,342 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:54:37,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:54:37,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:54:37,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8qt6.c.gcp-upstream.internal','N') ;"
2026-09-25 14:54:38,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:54:38,345 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:54:38,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:54:38,610 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:54:38,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:54:38,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:54:39,121 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:54:39,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:54:40.096 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.112 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.128 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.143 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.159 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.175 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.229 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.236 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.250 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.259 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.269 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.283 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.292 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.299 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.306 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40.312 UTC [32240] WARNING: there is no transaction in progress
2026-09-25 14:54:40,946 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:54:40,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:54:41,164 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:54:41,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:54:41,390 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:54:41,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:54:41,620 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:54:41,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:54:41,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:54:42,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8qt6.c.gcp-upstream.internal';"
2026-09-25 14:54:42,325 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:54:42,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:54:42,567 [I] DB_PATCHES have already been applied
I20260925 14:54:42.573101 21276 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyT1hpU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:54:42.573164 21276 mini_ranger.cc:193] Using host: 127.20.199.20
I20260925 14:54:42.576102 21276 mini_ranger.cc:227] Ranger admin URL: http://127.20.199.20:45173
Sep 25, 2026 2:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:54:43.722457 21276 mini_ranger.cc:160] Time spent starting Ranger: real 8.346s user 0.000s sys 0.008s
Sep 25, 2026 2:55:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:55:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:55:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:55:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:55:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:55:04.079685 21276 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/rangerkms_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/rangerkms_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.199.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.199.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:55:04 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:55:04.166651 21276 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:55:04.175952 21276 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:55:05.026882 21276 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:55:05.026983 21276 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms
I20260925 14:55:05.027004 21276 mini_ranger_kms.cc:227] Using postgres at 127.20.199.20:56779
2026-09-25 14:55:05,189 [I] DB FLAVOR :POSTGRES
2026-09-25 14:55:05,189 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:55:05,189 [I] Checking connection
2026-09-25 14:55:05,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:55:05,435 [I] connection success
2026-09-25 14:55:05,435 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:55:05,435 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:55:05,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:55:05,687 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:55:05,687 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:55:05,687 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:55:05,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:56779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:55:05,933 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:55:06.370255 21276 mini_ranger_kms.cc:310] Created kms service
I20260925 14:55:06.549190 21276 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:55:06.594364 21276 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:55:06 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348104, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:06.982002 21276 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:55:06.982115 21276 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:55:06.982149 21276 mini_ranger_kms.cc:240] Using host: 127.20.199.20
I20260925 14:55:06.985311 21276 mini_ranger_kms.cc:276] Ranger KMS PID: 32420
I20260925 14:55:06.985453 21276 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.199.20:39631
Sep 25, 2026 2:55:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:55:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:55:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:55:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:55:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:55:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 14:55:09.235828 21276 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.069s user 0.003s sys 0.142s
I20260925 14:55:09.235929 21276 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:55:09.236006 21276 mini_ranger_kms.cc:401] 127.20.199.20:39631/kms/v1/keys
Sep 25 14:55:09 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.199.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 14:55:09 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.199.20@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
Sep 25 14:55:10 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.199.20@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
Sep 25 14:55:12 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:55:12.398500 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42389
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:42389
--ranger_config_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:39631/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:12.477751 32471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:12.477980 32471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:12.478014 32471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:12.479121 32471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:12.479188 32471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:12.479212 32471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:12.479250 32471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:12.479297 32471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:12.480783 32471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.199.20:39631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:42389
--ranger_config_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:12.481046 32471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:12.481271 32471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:12.486419 32476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:12.486491 32471 server_base.cc:1034] running on GCE node
W20260925 14:55:12.486431 32479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:12.486433 32477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:12.486912 32471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:12.487244 32471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:12.488427 32471 hybrid_clock.cc:648] HybridClock initialized: now 1790348112488402 us; error 39 us; skew 500 ppm
Sep 25 14:55:12 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:12.491017 32471 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:55:12.491428 32471 webserver.cc:460] Webserver started at http://127.20.199.62:42397/ using document root <none> and password file <none>
I20260925 14:55:12.491659 32471 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:12.491719 32471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:12.491878 32471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:12 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:12.538857 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "075fb3c73d73414ba75a7391d45fe1a3"
format_stamp: "Formatted at 2026-09-25 14:55:12 on dist-test-slave-8qt6"
server_key: "6d48e58d81eb5436a1a221ce8d4179a7"
server_key_iv: "3587078ba403827a199ca9ce46756aab"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:12.539283 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "075fb3c73d73414ba75a7391d45fe1a3"
format_stamp: "Formatted at 2026-09-25 14:55:12 on dist-test-slave-8qt6"
server_key: "6d48e58d81eb5436a1a221ce8d4179a7"
server_key_iv: "3587078ba403827a199ca9ce46756aab"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:12.540714 32471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:12.555675 32486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:12.555927 32471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:12.556044 32471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "075fb3c73d73414ba75a7391d45fe1a3"
format_stamp: "Formatted at 2026-09-25 14:55:12 on dist-test-slave-8qt6"
server_key: "6d48e58d81eb5436a1a221ce8d4179a7"
server_key_iv: "3587078ba403827a199ca9ce46756aab"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:12.556174 32471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:12.587097 32471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:12.589205 32471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:12.589381 32471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:12.593652 32471 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:42389
I20260925 14:55:12.593708 32538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:42389 every 8 connection(s)
I20260925 14:55:12.594061 32471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:12.594369 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 32471
I20260925 14:55:12.594470 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:55:12.594641 21276 external_mini_cluster.cc:1428] Setting key 4762cfa7abc17e1c8b880be4a76b538d
I20260925 14:55:12.594869 32539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:12.598202 32539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3: Bootstrap starting.
I20260925 14:55:12.598821 32539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:12.599134 32539 log.cc:824] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:12.599857 32539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3: No bootstrap required, opened a new log
Sep 25 14:55:12 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:12.602066 32539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075fb3c73d73414ba75a7391d45fe1a3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42389 } }
I20260925 14:55:12.602196 32539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:12.602219 32539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075fb3c73d73414ba75a7391d45fe1a3, State: Initialized, Role: FOLLOWER
I20260925 14:55:12.602355 32539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075fb3c73d73414ba75a7391d45fe1a3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42389 } }
I20260925 14:55:12.602430 32539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:12.602452 32539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:12.602519 32539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:12.603046 32539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075fb3c73d73414ba75a7391d45fe1a3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42389 } }
I20260925 14:55:12.603127 32539 leader_election.cc:302] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 075fb3c73d73414ba75a7391d45fe1a3; no voters:
I20260925 14:55:12.603331 32539 leader_election.cc:288] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:12.603365 32544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:12.603477 32544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [term 1 LEADER]: Becoming Leader. State: Replica: 075fb3c73d73414ba75a7391d45fe1a3, State: Running, Role: LEADER
I20260925 14:55:12.603615 32539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:12.603618 32544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075fb3c73d73414ba75a7391d45fe1a3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42389 } }
I20260925 14:55:12.604081 32546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 075fb3c73d73414ba75a7391d45fe1a3. Latest consensus state: current_term: 1 leader_uuid: "075fb3c73d73414ba75a7391d45fe1a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075fb3c73d73414ba75a7391d45fe1a3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42389 } } }
I20260925 14:55:12.604171 32546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:12.604424 32545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "075fb3c73d73414ba75a7391d45fe1a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075fb3c73d73414ba75a7391d45fe1a3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42389 } } }
I20260925 14:55:12.604732 32545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:12.604879 32551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:12.605146 32539 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:55:12.605464 32551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:12.609172 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:12.595138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.20.199.62:42389)
0925 14:55:12.595345 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:12.595354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:12.595386 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:55:12.596027 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:12.596036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:12.596101 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:12.596273 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:12.596852 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:12.598260 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:12.599124 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:12.599699 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:12.599808 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:12.602374 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:12.602377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:12.602378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:12.602397 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:12.604351 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:12.604808 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:12.604811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:12.604812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:12.604912 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:12.605117 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:12.605118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:12.605119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:12.605446 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:12.607649 (+ 2203us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:12.607934 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 14:55:12.608658 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260925 14:55:12.611935 32551 catalog_manager.cc:1347] Generated new cluster ID: bda76f575d0d4f229d9909fb3273d5ad
I20260925 14:55:12.611990 32551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:12.628005 32551 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:12.628765 32551 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:12.638799 32551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3: Generated new TSK 0
I20260925 14:55:12.639004 32551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:55:13.263898 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:39631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42389
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:13.371168 32584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.371362 32584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.371384 32584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.372520 32584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.372563 32584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.372588 32584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:55:13.374132 32584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.199.20:39631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:13.374367 32584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.374610 32584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.380532 32590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.380532 32589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.380625 32592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.380952 32584 server_base.cc:1034] running on GCE node
I20260925 14:55:13.385394 32584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.385717 32584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.386876 32584 hybrid_clock.cc:648] HybridClock initialized: now 1790348113386865 us; error 33 us; skew 500 ppm
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.389712 32584 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:55:13.390308 32584 webserver.cc:460] Webserver started at http://127.20.199.1:37707/ using document root <none> and password file <none>
I20260925 14:55:13.390590 32584 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.390662 32584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.390822 32584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:13.404277 32584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "83a70859aa47400d878b32933a346785"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "4f2b6142b0369706b3925599e0993642"
server_key_iv: "1fe2a3b92dcfa468e13d8f525081b1b0"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.404834 32584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "83a70859aa47400d878b32933a346785"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "4f2b6142b0369706b3925599e0993642"
server_key_iv: "1fe2a3b92dcfa468e13d8f525081b1b0"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.406574 32584 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:55:13.420825 32599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.421012 32584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:13.421099 32584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "83a70859aa47400d878b32933a346785"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "4f2b6142b0369706b3925599e0993642"
server_key_iv: "1fe2a3b92dcfa468e13d8f525081b1b0"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.421201 32584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.443104 32584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.445722 32584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.445972 32584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.446260 32584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:13.446638 32584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:13.446684 32584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.446715 32584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:13.446736 32584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.453563 32584 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:43057
I20260925 14:55:13.453879 32714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:43057 every 8 connection(s)
I20260925 14:55:13.454008 32584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:13.461686 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 32584
I20260925 14:55:13.461808 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:13.462026 21276 external_mini_cluster.cc:1428] Setting key 65014b689a1cbd2c99b87fb3cab31c68
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:13.468014 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.454720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42389 (local address 127.20.199.1:38565)
0925 14:55:13.454891 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.454935 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.455463 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.458460 (+ 2997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.458471 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.458729 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.459051 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.459057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.459814 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.459816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.460728 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.460730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.461141 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.461141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.461289 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.461951 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.461962 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.463955 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.466200 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.466207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.466210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.466522 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.466755 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.466756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.466757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.466903 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.467229 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.467236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.467415 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.467532 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.467708 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260925 14:55:13.468531 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.454764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38565 (local address 127.20.199.62:42389)
0925 14:55:13.458222 (+ 3458us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.458224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.458233 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:13.458257 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.458258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.458268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.458359 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.459148 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.459722 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.460832 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.461048 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.461141 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.464063 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.464065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.464065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.464074 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.466108 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.466603 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.466604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.466605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.466684 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.466984 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.466985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.466985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.467135 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.467201 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.468302 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.468389 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3429,"thread_start_us":89,"threads_started":1}
I20260925 14:55:13.469152 32715 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42389
I20260925 14:55:13.469255 32715 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:13.469578 32715 heartbeater.cc:511] Master 127.20.199.62:42389 requested a full tablet report, sending...
I20260925 14:55:13.470156 32503 ts_manager.cc:194] Registered new tserver with Master: 83a70859aa47400d878b32933a346785 (127.20.199.1:43057)
I20260925 14:55:13.471076 32503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:38565
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:55:13.492462 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:39631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42389
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:13.597836 32721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.598052 32721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.598091 32721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.599304 32721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.599373 32721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.599418 32721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:55:13.600958 32721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.199.20:39631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:13.601215 32721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.601528 32721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.606999 32728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.607218 32729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.607453 32731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.608367 32721 server_base.cc:1034] running on GCE node
I20260925 14:55:13.608572 32721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.608855 32721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.610005 32721 hybrid_clock.cc:648] HybridClock initialized: now 1790348113609982 us; error 41 us; skew 500 ppm
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.612562 32721 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:55:13.613097 32721 webserver.cc:460] Webserver started at http://127.20.199.2:36635/ using document root <none> and password file <none>
I20260925 14:55:13.613425 32721 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.613498 32721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.613646 32721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:13.627177 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "fd6d8466029c4bc9a492db5685c8ec95"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "f9aeba7ce6d27a359651cc65b9517a36"
server_key_iv: "60fe588861bf0817301e54ddf3105942"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.627588 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "fd6d8466029c4bc9a492db5685c8ec95"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "f9aeba7ce6d27a359651cc65b9517a36"
server_key_iv: "60fe588861bf0817301e54ddf3105942"
server_key_version: "kuduclusterkey@0"
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.633651 32721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260925 14:55:13.672214 32738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.672531 32721 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.000s sys 0.000s
I20260925 14:55:13.672662 32721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "fd6d8466029c4bc9a492db5685c8ec95"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "f9aeba7ce6d27a359651cc65b9517a36"
server_key_iv: "60fe588861bf0817301e54ddf3105942"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.672796 32721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.712428 32721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.714999 32721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.715233 32721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.715533 32721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:13.716042 32721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:13.716125 32721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.716193 32721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:13.716251 32721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.722616 32721 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:43365
I20260925 14:55:13.723120 32721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:13.726105 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 32721
I20260925 14:55:13.726203 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:13.726399 21276 external_mini_cluster.cc:1428] Setting key d3849056ccf8501fbc7be64f937b501c
I20260925 14:55:13.728951 384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:43365 every 8 connection(s)
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
I20260925 14:55:13.753867 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.729999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:57153 (local address 127.20.199.62:42389)
0925 14:55:13.730043 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.730062 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.735198 (+ 5136us) server_negotiation.cc:407] Connection header received
0925 14:55:13.740224 (+ 5026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.740227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.740248 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.740283 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.741217 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.742054 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.743600 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.743913 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.743959 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.748641 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.748642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.748643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.748664 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.750449 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.750975 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.750976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.750977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.751035 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.752971 (+ 1936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.752973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.752973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.753120 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.753168 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.753642 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.753766 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:13.754891 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.729716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42389 (local address 127.20.199.2:57153)
0925 14:55:13.735007 (+ 5291us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.735090 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.735840 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.740537 (+ 4697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.740542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.740802 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.741095 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.741102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.742182 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.742182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.743495 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.743498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.744057 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.744058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.744204 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.744933 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.744944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.748515 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.750549 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.750557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.750560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.750880 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.752721 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.752722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.752723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.752883 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.753220 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.753226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.753471 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.754433 (+ 962us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.754622 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5231,"thread_start_us":87,"threads_started":1}
I20260925 14:55:13.755579 385 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42389
I20260925 14:55:13.755687 385 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:13.755882 385 heartbeater.cc:511] Master 127.20.199.62:42389 requested a full tablet report, sending...
I20260925 14:55:13.756304 32503 ts_manager.cc:194] Registered new tserver with Master: fd6d8466029c4bc9a492db5685c8ec95 (127.20.199.2:43365)
I20260925 14:55:13.756822 32503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:57153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:55:13.774411 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:39631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42389
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:13.902169 390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.902456 390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.902493 390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.904175 390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.904300 390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.904372 390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:55:13.906467 390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.199.20:39631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:13.906849 390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.907120 390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.913318 396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.913538 398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.913344 395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.914080 390 server_base.cc:1034] running on GCE node
I20260925 14:55:13.914299 390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.914547 390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.917869 390 hybrid_clock.cc:648] HybridClock initialized: now 1790348113917862 us; error 43 us; skew 500 ppm
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.920406 390 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:55:13.920926 390 webserver.cc:460] Webserver started at http://127.20.199.3:35681/ using document root <none> and password file <none>
I20260925 14:55:13.921164 390 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.921232 390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.921372 390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:13 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:13.959105 390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "be629cfe4d064d2cae0d01a096362bdd"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "5cfbb7230c2bad1ae06d934c29f832a5"
server_key_iv: "052f7fd4b8089a90ad534ec66b2f69ab"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.959590 390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "be629cfe4d064d2cae0d01a096362bdd"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "5cfbb7230c2bad1ae06d934c29f832a5"
server_key_iv: "052f7fd4b8089a90ad534ec66b2f69ab"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.961225 390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 14:55:13.980321 405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.980974 390 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 14:55:13.981096 390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "be629cfe4d064d2cae0d01a096362bdd"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-8qt6"
server_key: "5cfbb7230c2bad1ae06d934c29f832a5"
server_key_iv: "052f7fd4b8089a90ad534ec66b2f69ab"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:13.981206 390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.998723 390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:14.001107 390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:14.001322 390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:14.001674 390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:14.002072 390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:14.002171 390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:14.002239 390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:14.002293 390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:14.008558 390 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:36133
I20260925 14:55:14.009024 390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:14.011956 518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:36133 every 8 connection(s)
I20260925 14:55:14.018206 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 390
I20260925 14:55:14.018307 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:14.018484 21276 external_mini_cluster.cc:1428] Setting key 76d19d0926018730ca47b96603d2188f
Sep 25 14:55:14 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:14.023867 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.012688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:42445 (local address 127.20.199.62:42389)
0925 14:55:14.012753 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.012755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.013286 (+ 531us) server_negotiation.cc:407] Connection header received
0925 14:55:14.013959 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.013960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.013973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.014004 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.014818 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.015562 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.016971 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.017256 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.017297 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.020683 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.020685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.020685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.020693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.022421 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.022870 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.022871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.022871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.022925 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.023189 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.023190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.023190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.023311 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.023350 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.023696 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.023781 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:55:14.024595 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.012675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42389 (local address 127.20.199.3:42445)
0925 14:55:14.013161 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.013204 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.013834 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.014195 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.014200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.014434 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:14.014709 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.014715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.015678 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.015679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.016871 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.016874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.017385 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.017386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.017558 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.018779 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:14.018789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:14.020576 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:14.022515 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.022522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.022524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.022785 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.022989 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:14.022990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:14.022990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:14.023106 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:14.023393 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:14.023399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:14.023551 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.024223 (+ 672us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.024378 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":93,"threads_started":1}
I20260925 14:55:14.025276 519 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42389
I20260925 14:55:14.025494 519 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:14.025683 519 heartbeater.cc:511] Master 127.20.199.62:42389 requested a full tablet report, sending...
I20260925 14:55:14.026059 32503 ts_manager.cc:194] Registered new tserver with Master: be629cfe4d064d2cae0d01a096362bdd (127.20.199.3:36133)
I20260925 14:55:14.026525 32503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:42445
I20260925 14:55:14.029989 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:14.039332 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.031755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.20.199.62:42389)
0925 14:55:14.031793 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.031795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.032066 (+ 271us) server_negotiation.cc:407] Connection header received
0925 14:55:14.032173 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.032174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.032184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.032211 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:14.032932 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.033710 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.034777 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.035049 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.035085 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.036547 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:14.036549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:14.036549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:14.036555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:14.038195 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.038561 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.038562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.038563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.038611 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:14.038796 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:14.038797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:14.038797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:14.038910 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:14.038978 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.039177 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.039247 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:55:14.041356 32503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:14.043488 32503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:14.060228 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.046217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:36133 (local address 127.0.0.1:45430)
0925 14:55:14.047264 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.047304 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.047363 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.049919 (+ 2556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.049923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.049926 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:14.049991 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.049993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.055043 (+ 5050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.055043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.056344 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.056347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.059904 (+ 3557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.059904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.059990 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.060003 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.060077 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.060139 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":69,"threads_started":1}
I20260925 14:55:14.060294 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.046450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.20.199.3:36133)
0925 14:55:14.049676 (+ 3226us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.049680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.049690 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:14.049729 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.049731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.049748 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.049830 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:14.050153 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.054960 (+ 4807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.057782 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.059686 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.059757 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.059845 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.060159 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.060208 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3191,"thread_start_us":69,"threads_started":1}
I20260925 14:55:14.061530 453 tablet_service.cc:1467] Processing CreateTablet for tablet ea6d6c2781be4e12beea315b36fa5bf4 (DEFAULT_TABLE table=test-table [id=1fbeacd109e34787b1ac1e3a84eb6f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:14.061909 453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6d6c2781be4e12beea315b36fa5bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:14.063731 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.046919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:43365 (local address 127.0.0.1:33578)
0925 14:55:14.051729 (+ 4810us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.051738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.051780 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.052023 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.052023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.052025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:14.052077 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.052079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.056440 (+ 4361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.056440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.062471 (+ 6031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.062474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.063476 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.063477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.063551 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.063556 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.063609 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.063651 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4796,"thread_start_us":62,"threads_started":1}
I20260925 14:55:14.064006 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.047052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.20.199.2:43365)
0925 14:55:14.049589 (+ 2537us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.049594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.051833 (+ 2239us) server_negotiation.cc:407] Connection header received
0925 14:55:14.051874 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.051876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.051890 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.051952 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:14.052158 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.053218 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.062590 (+ 9372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.063304 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.063373 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.063439 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.063882 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.063926 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2495,"thread_start_us":74,"threads_started":1}
I20260925 14:55:14.064493 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.046752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43057 (local address 127.0.0.1:55616)
0925 14:55:14.051410 (+ 4658us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.051420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.051466 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.053311 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.053312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.053314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:14.053351 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.053352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.057573 (+ 4221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.057573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.061012 (+ 3439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.061015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.064270 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.064271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.064332 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.064336 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.064389 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.064426 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4642,"mutex_wait_us":1100,"thread_start_us":83,"threads_started":1}
I20260925 14:55:14.064754 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.046842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.20.199.1:43057)
0925 14:55:14.050107 (+ 3265us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.050111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.051523 (+ 1412us) server_negotiation.cc:407] Connection header received
0925 14:55:14.051561 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.051563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.051578 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.051644 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:14.053446 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.054779 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.061108 (+ 6329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.062247 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.062318 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.062371 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.064628 (+ 2257us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.064685 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3233,"thread_start_us":46,"threads_started":1}
I20260925 14:55:14.064656 317 tablet_service.cc:1467] Processing CreateTablet for tablet ea6d6c2781be4e12beea315b36fa5bf4 (DEFAULT_TABLE table=test-table [id=1fbeacd109e34787b1ac1e3a84eb6f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:14.064982 317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6d6c2781be4e12beea315b36fa5bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:14.065145 537 tablet_bootstrap.cc:492] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Bootstrap starting.
I20260925 14:55:14.066149 537 tablet_bootstrap.cc:654] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:14.066473 537 log.cc:824] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:14.067399 537 tablet_bootstrap.cc:492] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: No bootstrap required, opened a new log
I20260925 14:55:14.067469 537 ts_tablet_manager.cc:1397] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:14.068289 539 tablet_bootstrap.cc:492] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Bootstrap starting.
I20260925 14:55:14.068557 32649 tablet_service.cc:1467] Processing CreateTablet for tablet ea6d6c2781be4e12beea315b36fa5bf4 (DEFAULT_TABLE table=test-table [id=1fbeacd109e34787b1ac1e3a84eb6f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:14.068930 32649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6d6c2781be4e12beea315b36fa5bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:14.069249 537 raft_consensus.cc:348] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.069458 537 raft_consensus.cc:374] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:14.069500 537 raft_consensus.cc:729] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be629cfe4d064d2cae0d01a096362bdd, State: Initialized, Role: FOLLOWER
I20260925 14:55:14.069605 537 consensus_queue.cc:260] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.069861 537 ts_tablet_manager.cc:1428] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:55:14.069998 519 heartbeater.cc:503] Master 127.20.199.62:42389 was elected leader, sending a full tablet report...
I20260925 14:55:14.072646 539 tablet_bootstrap.cc:654] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:14.073004 539 log.cc:824] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:14.073493 541 tablet_bootstrap.cc:492] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: Bootstrap starting.
I20260925 14:55:14.074179 541 tablet_bootstrap.cc:654] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:14.074501 541 log.cc:824] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:14.075284 539 tablet_bootstrap.cc:492] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: No bootstrap required, opened a new log
I20260925 14:55:14.075389 539 ts_tablet_manager.cc:1397] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 14:55:14.077338 539 raft_consensus.cc:348] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.077610 539 raft_consensus.cc:374] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:14.077684 539 raft_consensus.cc:729] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6d8466029c4bc9a492db5685c8ec95, State: Initialized, Role: FOLLOWER
I20260925 14:55:14.077847 539 consensus_queue.cc:260] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.078136 539 ts_tablet_manager.cc:1428] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:55:14.078275 385 heartbeater.cc:503] Master 127.20.199.62:42389 was elected leader, sending a full tablet report...
I20260925 14:55:14.082090 541 tablet_bootstrap.cc:492] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: No bootstrap required, opened a new log
I20260925 14:55:14.082168 541 ts_tablet_manager.cc:1397] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260925 14:55:14.084754 541 raft_consensus.cc:348] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.084925 541 raft_consensus.cc:374] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:14.084956 541 raft_consensus.cc:729] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a70859aa47400d878b32933a346785, State: Initialized, Role: FOLLOWER
I20260925 14:55:14.085239 541 consensus_queue.cc:260] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.085569 541 ts_tablet_manager.cc:1428] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:14.085739 32715 heartbeater.cc:503] Master 127.20.199.62:42389 was elected leader, sending a full tablet report...
Sep 25 14:55:14 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
W20260925 14:55:14.204910 32716 tablet.cc:2369] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:14.229897 386 tablet.cc:2369] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:14.263339 521 tablet.cc:2369] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:14.315050 544 raft_consensus.cc:482] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:14.315194 544 raft_consensus.cc:504] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.315575 544 leader_election.cc:288] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be629cfe4d064d2cae0d01a096362bdd (127.20.199.3:36133), 83a70859aa47400d878b32933a346785 (127.20.199.1:43057)
I20260925 14:55:14.320178 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.316068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:48709 (local address 127.20.199.1:43057)
0925 14:55:14.316119 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.316121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.316479 (+ 358us) server_negotiation.cc:407] Connection header received
0925 14:55:14.316582 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.316583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.316593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.316623 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:14.316853 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.317613 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.319138 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.319726 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.319778 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.319787 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.320023 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.320075 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:55:14.320652 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.315871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43057 (local address 127.20.199.2:48709)
0925 14:55:14.316419 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.316429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.316508 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.316709 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.316710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.316712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:14.316780 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.316782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.317750 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.317751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.319039 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.319042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.319850 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.319851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.319937 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.319942 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.320519 (+ 577us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.320565 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":71,"threads_started":1}
I20260925 14:55:14.321131 32663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6d6c2781be4e12beea315b36fa5bf4" candidate_uuid: "fd6d8466029c4bc9a492db5685c8ec95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a70859aa47400d878b32933a346785" is_pre_election: true
I20260925 14:55:14.321292 32663 raft_consensus.cc:2454] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6d8466029c4bc9a492db5685c8ec95 in term 0.
I20260925 14:55:14.321633 32740 leader_election.cc:302] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a70859aa47400d878b32933a346785, fd6d8466029c4bc9a492db5685c8ec95; no voters:
I20260925 14:55:14.321781 544 raft_consensus.cc:2781] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:14.321820 544 raft_consensus.cc:482] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:14.321842 544 raft_consensus.cc:3037] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:14.322847 544 raft_consensus.cc:504] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.322995 544 leader_election.cc:288] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [CANDIDATE]: Term 1 election: Requested vote from peers be629cfe4d064d2cae0d01a096362bdd (127.20.199.3:36133), 83a70859aa47400d878b32933a346785 (127.20.199.1:43057)
I20260925 14:55:14.323308 32663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6d6c2781be4e12beea315b36fa5bf4" candidate_uuid: "fd6d8466029c4bc9a492db5685c8ec95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a70859aa47400d878b32933a346785"
I20260925 14:55:14.323388 32663 raft_consensus.cc:3037] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:14.324261 32663 raft_consensus.cc:2454] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6d8466029c4bc9a492db5685c8ec95 in term 1.
I20260925 14:55:14.324476 32740 leader_election.cc:302] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a70859aa47400d878b32933a346785, fd6d8466029c4bc9a492db5685c8ec95; no voters:
I20260925 14:55:14.324586 544 raft_consensus.cc:2781] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:14.324740 544 raft_consensus.cc:686] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 1 LEADER]: Becoming Leader. State: Replica: fd6d8466029c4bc9a492db5685c8ec95, State: Running, Role: LEADER
I20260925 14:55:14.324847 544 consensus_queue.cc:237] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } }
I20260925 14:55:14.325858 32503 catalog_manager.cc:5495] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd6d8466029c4bc9a492db5685c8ec95 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "fd6d8466029c4bc9a492db5685c8ec95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd6d8466029c4bc9a492db5685c8ec95" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 43365 } health_report { overall_health: HEALTHY } } }
I20260925 14:55:14.330089 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.316365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:42821 (local address 127.20.199.3:36133)
0925 14:55:14.316426 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.316428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.326701 (+ 10273us) server_negotiation.cc:407] Connection header received
0925 14:55:14.326809 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.326810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.326822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.326852 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:14.327068 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.327762 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.329078 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.329699 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.329754 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.329763 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.329977 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.330019 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:55:14.330538 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.316313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:36133 (local address 127.20.199.2:42821)
0925 14:55:14.326620 (+ 10307us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.326633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:14.326730 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:14.326921 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:14.326922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:14.326923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:14.326991 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.326992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.327889 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.327890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.328967 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:14.328970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.329823 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.329824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:14.329895 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.329899 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:14.330361 (+ 462us) client_negotiation.cc:240] Negotiation successful
0925 14:55:14.330440 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10291,"thread_start_us":52,"threads_started":1}
I20260925 14:55:14.331280 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6d6c2781be4e12beea315b36fa5bf4" candidate_uuid: "fd6d8466029c4bc9a492db5685c8ec95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be629cfe4d064d2cae0d01a096362bdd" is_pre_election: true
I20260925 14:55:14.331426 470 raft_consensus.cc:2454] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6d8466029c4bc9a492db5685c8ec95 in term 0.
I20260925 14:55:14.331545 469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6d6c2781be4e12beea315b36fa5bf4" candidate_uuid: "fd6d8466029c4bc9a492db5685c8ec95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be629cfe4d064d2cae0d01a096362bdd"
I20260925 14:55:14.331604 469 raft_consensus.cc:3037] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:14.332839 469 raft_consensus.cc:2454] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6d8466029c4bc9a492db5685c8ec95 in term 1.
Sep 25 14:55:14 dist-test-slave-8qt6 krb5kdc[32027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:14.401831 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.398266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.20.199.2:43365)
0925 14:55:14.398325 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:14.398327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:14.398336 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:14.398378 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:14.398379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:14.398401 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:14.398473 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:14.398673 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.399411 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.400549 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:14.400888 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:14.400960 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:14.401188 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:14.401323 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:14.401421 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:14.401692 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:55:14.401719 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:55:14.405040 469 raft_consensus.cc:1261] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 1 FOLLOWER]: Refusing update from remote peer fd6d8466029c4bc9a492db5685c8ec95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:14.405431 32663 raft_consensus.cc:1261] T ea6d6c2781be4e12beea315b36fa5bf4 P 83a70859aa47400d878b32933a346785 [term 1 FOLLOWER]: Refusing update from remote peer fd6d8466029c4bc9a492db5685c8ec95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:14.405809 544 consensus_queue.cc:1035] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a70859aa47400d878b32933a346785" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:14.405980 548 consensus_queue.cc:1035] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be629cfe4d064d2cae0d01a096362bdd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:14.410362 553 mvcc.cc:205] Tried to move back new op lower bound from 7333265876596166656 to 7333265876275134464. Current Snapshot: MvccSnapshot[applied={T|T < 7333265876596166656}]
I20260925 14:55:14.410609 552 mvcc.cc:205] Tried to move back new op lower bound from 7333265876596166656 to 7333265876275134464. Current Snapshot: MvccSnapshot[applied={T|T < 7333265876596166656}]
I20260925 14:55:14.412971 555 mvcc.cc:205] Tried to move back new op lower bound from 7333265876596166656 to 7333265876275134464. Current Snapshot: MvccSnapshot[applied={T|T < 7333265876596166656}]
I20260925 14:55:14.416005 32503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:14.416102 32503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:14.416934 32503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 075fb3c73d73414ba75a7391d45fe1a3: Sending DeleteTablet for 3 replicas of tablet ea6d6c2781be4e12beea315b36fa5bf4
I20260925 14:55:14.417481 317 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6d6c2781be4e12beea315b36fa5bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:14 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:33578
I20260925 14:55:14.417855 453 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6d6c2781be4e12beea315b36fa5bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:14 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:45430
I20260925 14:55:14.417418 32649 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6d6c2781be4e12beea315b36fa5bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:14 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:55616
I20260925 14:55:14.418692 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 32584
I20260925 14:55:14.425086 562 tablet_replica.cc:330] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: stopping tablet replica
I20260925 14:55:14.425300 562 raft_consensus.cc:2229] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:14.425536 562 raft_consensus.cc:2258] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:14.426440 561 tablet_replica.cc:330] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: stopping tablet replica
I20260925 14:55:14.426613 561 raft_consensus.cc:2229] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:14.426779 561 raft_consensus.cc:2258] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 14:55:14.427245 32488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:43057 (error 108)
I20260925 14:55:14.427525 561 ts_tablet_manager.cc:1905] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:14.429294 561 ts_tablet_manager.cc:1918] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:14.429477 561 log.cc:1196] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/ea6d6c2781be4e12beea315b36fa5bf4
I20260925 14:55:14.429828 561 ts_tablet_manager.cc:1939] T ea6d6c2781be4e12beea315b36fa5bf4 P fd6d8466029c4bc9a492db5685c8ec95: Deleting consensus metadata
I20260925 14:55:14.430337 32488 catalog_manager.cc:4841] TS fd6d8466029c4bc9a492db5685c8ec95 (127.20.199.2:43365): tablet ea6d6c2781be4e12beea315b36fa5bf4 (table test-table [id=1fbeacd109e34787b1ac1e3a84eb6f39]) successfully deleted
I20260925 14:55:14.430833 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 32721
I20260925 14:55:14.431453 562 ts_tablet_manager.cc:1905] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:14.435259 562 ts_tablet_manager.cc:1918] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:14.435340 562 log.cc:1196] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/ea6d6c2781be4e12beea315b36fa5bf4
I20260925 14:55:14.435612 562 ts_tablet_manager.cc:1939] T ea6d6c2781be4e12beea315b36fa5bf4 P be629cfe4d064d2cae0d01a096362bdd: Deleting consensus metadata
I20260925 14:55:14.436228 32489 catalog_manager.cc:4841] TS be629cfe4d064d2cae0d01a096362bdd (127.20.199.3:36133): tablet ea6d6c2781be4e12beea315b36fa5bf4 (table test-table [id=1fbeacd109e34787b1ac1e3a84eb6f39]) successfully deleted
I20260925 14:55:14.436585 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:14.436440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43057 (local address 127.0.0.1:55628)
0925 14:55:14.436501 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:14.436544 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 14:55:14.436698 32488 catalog_manager.cc:4584] TS 83a70859aa47400d878b32933a346785 (127.20.199.1:43057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea6d6c2781be4e12beea315b36fa5bf4: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43057: connect: Connection refused (error 111)
I20260925 14:55:14.438581 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 390
I20260925 14:55:14.444736 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 32471
I20260925 14:55:14.449779 21276 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:55:14.794236 21276 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:55:14.794353 21276 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:55:15.138871 21276 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:55:15.139 UTC [32057] LOG: received smart shutdown request
2026-09-25 14:55:15.142 UTC [32057] LOG: background worker "logical replication launcher" (PID 32066) exited with exit code 1
2026-09-25 14:55:15.143 UTC [32061] LOG: shutting down
2026-09-25 14:55:15.312 UTC [32057] LOG: database system is shut down
2026-09-25T14:55:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44724 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:15 dist-test-slave-8qt6 krb5kdc[574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:15 dist-test-slave-8qt6 krb5kdc[574](info): set up 2 sockets
Sep 25 14:55:15 dist-test-slave-8qt6 krb5kdc[574](info): commencing operation
krb5kdc: starting...
W20260925 14:55:16.469542 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:55:16 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/rangeradmin_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/rangeradmin_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/rangerlookup_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/rangerlookup_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
I20260925 14:55:16.636761 21276 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyT1hpU/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/postgres -l logfile start
2026-09-25 14:55:19.218 UTC [604] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:55:19.218 UTC [604] LOG: listening on IPv4 address "127.20.199.20", port 44319
2026-09-25 14:55:19.222 UTC [604] LOG: listening on Unix socket "/tmp/.s.PGSQL.44319"
2026-09-25 14:55:19.245 UTC [607] LOG: database system was shut down at 2026-09-25 14:55:17 UTC
2026-09-25 14:55:19.249 UTC [604] LOG: database system is ready to accept connections
I20260925 14:55:20.201341 21276 mini_postgres.cc:96] Postgres bound to 44319
2026-09-25 14:55:20.207 UTC [615] FATAL: database "slave" does not exist
127.20.199.20:44319 - accepting connections
I20260925 14:55:20.207957 21276 mini_ranger.cc:161] Starting Ranger...
I20260925 14:55:20.223913 21276 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:55:20.744858 21276 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:55:20.744954 21276 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-admin
2026-09-25 14:55:21,015 [I] DB FLAVOR :POSTGRES
2026-09-25 14:55:21,016 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:55:21,016 [I] Checking connection..
2026-09-25 14:55:21,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:55:21,268 [I] Checking connection passed.
2026-09-25 14:55:21,268 [I] --------- Verifying version history table ---------
2026-09-25 14:55:21,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:55:21,513 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:55:21,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:55:21,763 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:55:21,763 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:55:21,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:55:21.997 UTC [679] WARNING: there is no transaction in progress
2026-09-25 14:55:22,003 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:55:22,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:55:22,263 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:55:22,264 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:55:22,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:55:22,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:55:22,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8qt6.c.gcp-upstream.internal','N') ;"
2026-09-25 14:55:22,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:55:23,214 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:55:23,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:55:23,440 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:55:23,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:55:23,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:55:23,946 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:55:23,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:55:24.924 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:24.941 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:24.958 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:24.974 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:24.991 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.009 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.075 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.082 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.096 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.105 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.116 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.129 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.138 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.144 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.151 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25.157 UTC [787] WARNING: there is no transaction in progress
2026-09-25 14:55:25,807 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:55:25,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:55:26,045 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:55:26,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:55:26,302 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:55:26,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:55:26,548 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:55:26,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:55:26,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:55:27,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8qt6.c.gcp-upstream.internal';"
2026-09-25 14:55:27,247 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:55:27,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:55:27,502 [I] DB_PATCHES have already been applied
I20260925 14:55:27.508095 21276 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyT1hpU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:55:27.508155 21276 mini_ranger.cc:193] Using host: 127.20.199.20
I20260925 14:55:27.511163 21276 mini_ranger.cc:227] Ranger admin URL: http://127.20.199.20:60043
Sep 25, 2026 2:55:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:55:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:55:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:55:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:55:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:55:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:55:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:55:28.640832 21276 mini_ranger.cc:160] Time spent starting Ranger: real 8.433s user 0.000s sys 0.008s
Sep 25, 2026 2:55:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:55:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:55:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:55:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:55:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:55:48.802330 21276 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.199.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.199.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/rangerkms_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.199.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/rangerkms_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.199.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.199.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.199.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/HTTP_127.20.199.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:55:48 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348148, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:55:48.896860 21276 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:55:48.908299 21276 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:55:49.817186 21276 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:55:49.817278 21276 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms
I20260925 14:55:49.817296 21276 mini_ranger_kms.cc:227] Using postgres at 127.20.199.20:44319
2026-09-25 14:55:50,024 [I] DB FLAVOR :POSTGRES
2026-09-25 14:55:50,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:55:50,024 [I] Checking connection
2026-09-25 14:55:50,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:55:50,309 [I] connection success
2026-09-25 14:55:50,309 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:55:50,309 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:55:50,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:55:50,587 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:55:50,587 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:55:50,587 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:55:50,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.199.20:44319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:55:50,868 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:55:51.306167 21276 mini_ranger_kms.cc:310] Created kms service
I20260925 14:55:51.537858 21276 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:55:51.580121 21276 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:55:51 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348148, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:51.995390 21276 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:55:51.995602 21276 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyT1hpU/build/release/bin/postgresql.jar:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyT1hpU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:55:51.995626 21276 mini_ranger_kms.cc:240] Using host: 127.20.199.20
I20260925 14:55:51.998805 21276 mini_ranger_kms.cc:276] Ranger KMS PID: 967
I20260925 14:55:51.998888 21276 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.199.20:53839
Sep 25, 2026 2:55:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:55:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:55:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:55:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:55:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:55:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 14:55:54.297747 21276 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.401s user 0.008s sys 0.181s
I20260925 14:55:54.297865 21276 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:55:54.297919 21276 mini_ranger_kms.cc:401] 127.20.199.20:53839/kms/v1/keys
Sep 25 14:55:54 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.199.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 14:55:54 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.199.20@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
Sep 25 14:55:55 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.199.20@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
Sep 25 14:55:57 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:55:57.440635 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46557
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:46557
--ranger_config_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:57.522076 1018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:57.522284 1018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:57.522306 1018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:57.523518 1018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:57.523568 1018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:57.523583 1018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:57.523595 1018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:57.523607 1018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:57.525326 1018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.199.20:53839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:46557
--ranger_config_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:57.525584 1018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:57.525816 1018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:57.531543 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:57.531524 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:57.531524 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:57.531559 1018 server_base.cc:1034] running on GCE node
I20260925 14:55:57.531978 1018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:57.532244 1018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:57.533486 1018 hybrid_clock.cc:648] HybridClock initialized: now 1790348157533477 us; error 41 us; skew 500 ppm
Sep 25 14:55:57 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:57.542236 1018 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:55:57.542735 1018 webserver.cc:460] Webserver started at http://127.20.199.62:45129/ using document root <none> and password file <none>
I20260925 14:55:57.542968 1018 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:57.543030 1018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:57.543200 1018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:57 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:57.600412 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "58809930c9ae4f1d9083f15cb11a373b"
format_stamp: "Formatted at 2026-09-25 14:55:57 on dist-test-slave-8qt6"
server_key: "b7fa6abaf93dcaf0a1f091cd992eb7f2"
server_key_iv: "d69350a2972ab655011afab1801ef6b1"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:57.600847 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "58809930c9ae4f1d9083f15cb11a373b"
format_stamp: "Formatted at 2026-09-25 14:55:57 on dist-test-slave-8qt6"
server_key: "b7fa6abaf93dcaf0a1f091cd992eb7f2"
server_key_iv: "d69350a2972ab655011afab1801ef6b1"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:57.602421 1018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 14:55:57.614310 1033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:57.614552 1018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:57.614655 1018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "58809930c9ae4f1d9083f15cb11a373b"
format_stamp: "Formatted at 2026-09-25 14:55:57 on dist-test-slave-8qt6"
server_key: "b7fa6abaf93dcaf0a1f091cd992eb7f2"
server_key_iv: "d69350a2972ab655011afab1801ef6b1"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:57.614773 1018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:57.640103 1018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:57.642403 1018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:57.642576 1018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:57.646649 1018 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:46557
I20260925 14:55:57.646687 1085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:46557 every 8 connection(s)
I20260925 14:55:57.647083 1018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:57.647799 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:57.650278 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Bootstrap starting.
I20260925 14:55:57.650882 1086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:57.651134 1086 log.cc:824] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:57.651841 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: No bootstrap required, opened a new log
I20260925 14:55:57.654001 1086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:57.654162 1086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:57.654197 1086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58809930c9ae4f1d9083f15cb11a373b, State: Initialized, Role: FOLLOWER
I20260925 14:55:57.654358 1086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:57.654449 1086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:57.654475 1086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:57.654516 1086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:57.655100 1086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:57.655191 1086 leader_election.cc:302] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58809930c9ae4f1d9083f15cb11a373b; no voters:
I20260925 14:55:57.655325 1086 leader_election.cc:288] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:57.655398 1089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:57.655575 1089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 LEADER]: Becoming Leader. State: Replica: 58809930c9ae4f1d9083f15cb11a373b, State: Running, Role: LEADER
I20260925 14:55:57.655653 1086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:57.655689 1089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:57.656194 1090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58809930c9ae4f1d9083f15cb11a373b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } } }
I20260925 14:55:57.656227 1091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58809930c9ae4f1d9083f15cb11a373b. Latest consensus state: current_term: 1 leader_uuid: "58809930c9ae4f1d9083f15cb11a373b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } } }
I20260925 14:55:57.656426 1091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:57.656613 1090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:57.656852 1098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:57.656872 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1018
I20260925 14:55:57.656967 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:55:57.657166 21276 external_mini_cluster.cc:1428] Setting key 9dd04090d317e0da8bdabbe7b3049dd8
I20260925 14:55:57.657490 1098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:57.657787 1086 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:55:57.662415 1098 catalog_manager.cc:1347] Generated new cluster ID: d7f4bd948122460abca5d6e91e9edb37
I20260925 14:55:57.662593 1098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:57 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:57.670313 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:57.657903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.20.199.62:46557)
0925 14:55:57.658244 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:57.658254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:57.658289 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:55:57.658410 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:57.658418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:57.658486 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:57.658636 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:57.660335 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.662833 (+ 2498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.663648 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.664167 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.664310 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:57.666705 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:57.666709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:57.666709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:57.666719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:57.668608 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:57.668979 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:57.668982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:57.668982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:57.669082 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:57.669310 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:57.669311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:57.669311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:57.669703 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:57.669884 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:57.669989 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:55:57.670116 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":75,"threads_started":1}
I20260925 14:55:57.698567 1098 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:57.699471 1098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:57.714510 1098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Generated new TSK 0
I20260925 14:55:57.714723 1098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:55:58.400034 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46557
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:58.487294 1131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.487519 1131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.487557 1131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.488673 1131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.488737 1131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.488786 1131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:55:58.490315 1131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.199.20:53839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:58.490566 1131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.490790 1131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.495853 1136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.495899 1137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.495900 1139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.496526 1131 server_base.cc:1034] running on GCE node
I20260925 14:55:58.496733 1131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.496984 1131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.498139 1131 hybrid_clock.cc:648] HybridClock initialized: now 1790348158498127 us; error 31 us; skew 500 ppm
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:58.500813 1131 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:55:58.501344 1131 webserver.cc:460] Webserver started at http://127.20.199.1:44341/ using document root <none> and password file <none>
I20260925 14:55:58.501621 1131 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.501691 1131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.501834 1131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:58.516912 1131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "0930153463d144dcbd7523c2939298a0"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "6e0ce810db00073250c9f30f51c71276"
server_key_iv: "1bb78b2ac4ea744ba5b778847effc6a6"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:58.517778 1131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "0930153463d144dcbd7523c2939298a0"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "6e0ce810db00073250c9f30f51c71276"
server_key_iv: "1bb78b2ac4ea744ba5b778847effc6a6"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:58.525204 1131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 14:55:58.540297 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.540567 1131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:58.540676 1131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "0930153463d144dcbd7523c2939298a0"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "6e0ce810db00073250c9f30f51c71276"
server_key_iv: "1bb78b2ac4ea744ba5b778847effc6a6"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:58.540771 1131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.581218 1131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.583768 1131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.583976 1131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.584250 1131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.584658 1131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.584736 1131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.584793 1131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.584826 1131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.591694 1131 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:43047
I20260925 14:55:58.592149 1131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:58.592628 1261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:43047 every 8 connection(s)
I20260925 14:55:58.595711 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1131
I20260925 14:55:58.595819 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:58.596047 21276 external_mini_cluster.cc:1428] Setting key 4426c23af12a2d187ae3d9257bed385c
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:58.604524 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.593456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46557 (local address 127.20.199.1:44157)
0925 14:55:58.593903 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.593949 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.594534 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.594901 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.594905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.595144 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.595427 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.595433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.596485 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.596487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.597779 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.597782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.598319 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.598320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.598472 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.599083 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:58.599094 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.600775 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.602837 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:58.602844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:58.602847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:58.603142 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:58.603377 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:58.603378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:58.603378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:58.603522 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:58.603831 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.603836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.604002 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.604105 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.604262 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":95,"threads_started":1}
I20260925 14:55:58.605297 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.593751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:44157 (local address 127.20.199.62:46557)
0925 14:55:58.594109 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.594112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.594126 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:55:58.594700 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.594701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.594713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.594776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.595537 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.596375 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.597877 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.598176 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.598240 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.600890 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.600893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.600893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:58.600901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.602740 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:58.603228 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:58.603230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:58.603230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:58.603293 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:58.603606 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:58.603607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:58.603608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:58.603748 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.603805 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.605116 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.605204 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":75,"threads_started":1}
I20260925 14:55:58.606191 1262 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46557
I20260925 14:55:58.606287 1262 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.606483 1262 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:55:58.607053 1050 ts_manager.cc:194] Registered new tserver with Master: 0930153463d144dcbd7523c2939298a0 (127.20.199.1:43047)
I20260925 14:55:58.607940 1050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:44157
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:55:58.637060 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46557
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:55:58.768950 1268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.769292 1268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.769418 1268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.771245 1268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.771374 1268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.771457 1268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:55:58.773641 1268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.199.20:53839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:58.774120 1268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.774458 1268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.780826 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.781015 1275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.781174 1268 server_base.cc:1034] running on GCE node
W20260925 14:55:58.781544 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.781810 1268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.782075 1268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.783288 1268 hybrid_clock.cc:648] HybridClock initialized: now 1790348158783268 us; error 31 us; skew 500 ppm
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:58.792660 1268 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:55:58.793190 1268 webserver.cc:460] Webserver started at http://127.20.199.2:45889/ using document root <none> and password file <none>
I20260925 14:55:58.793550 1268 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.793618 1268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.793792 1268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:58.807734 1268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "971f94ae769d4af7aa4a97150243e43d"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "eac6b1f10d80745b45c9f8c61402ca43"
server_key_iv: "fb22de64bf7b4ee74a39fb6bb5529795"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:58.808143 1268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "971f94ae769d4af7aa4a97150243e43d"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "eac6b1f10d80745b45c9f8c61402ca43"
server_key_iv: "fb22de64bf7b4ee74a39fb6bb5529795"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:58.809705 1268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:58.833570 1286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.833851 1268 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260925 14:55:58.833943 1268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "971f94ae769d4af7aa4a97150243e43d"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "eac6b1f10d80745b45c9f8c61402ca43"
server_key_iv: "fb22de64bf7b4ee74a39fb6bb5529795"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:58.834038 1268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.850178 1268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.852455 1268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.852609 1268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.852829 1268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.853190 1268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.853245 1268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.853281 1268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.853307 1268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.860829 1268 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44835
I20260925 14:55:58.861012 1399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44835 every 8 connection(s)
I20260925 14:55:58.861361 1268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:58.865552 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1268
I20260925 14:55:58.865662 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:58.865859 21276 external_mini_cluster.cc:1428] Setting key c0ec9bdb27aa5e716fe3d2ec3e28e069
Sep 25 14:55:58 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:58.873533 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.862177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:58447 (local address 127.20.199.62:46557)
0925 14:55:58.862227 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.862230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.862545 (+ 315us) server_negotiation.cc:407] Connection header received
0925 14:55:58.863181 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.863183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.863201 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.863286 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.863982 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.864765 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.866293 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.866605 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.866696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.869872 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.869874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.869875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:58.869882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.871793 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:58.872298 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:58.872300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:58.872301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:58.872369 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:58.872692 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:58.872693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:58.872693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:58.872862 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.872946 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.873279 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.873380 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:58.873667 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.862155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46557 (local address 127.20.199.2:58447)
0925 14:55:58.862394 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.862436 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.863059 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.863357 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.863361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.863594 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.863871 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.863877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.864893 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.864894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.866198 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.866201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.866715 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.866716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.866853 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.867425 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:58.867440 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.869737 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.871892 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:58.871899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:58.871901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:58.872208 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:58.872452 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:58.872458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:58.872458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:58.872602 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:58.872953 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.872959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.873129 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.873253 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.873435 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260925 14:55:58.874825 1400 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46557
I20260925 14:55:58.874933 1400 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.875131 1400 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:55:58.875578 1050 ts_manager.cc:194] Registered new tserver with Master: 971f94ae769d4af7aa4a97150243e43d (127.20.199.2:44835)
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:55:58.876121 1050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:58447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:55:58.892582 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46557
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:59.006371 1405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:59.006632 1405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:59.006675 1405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:59.008064 1405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:59.008131 1405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:59.008175 1405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:55:59.010344 1405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.199.20:53839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:59.010628 1405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:59.010874 1405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:59.016207 1411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.016284 1410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.016636 1413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:59.017458 1405 server_base.cc:1034] running on GCE node
I20260925 14:55:59.017652 1405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:59.017964 1405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:59.019141 1405 hybrid_clock.cc:648] HybridClock initialized: now 1790348159019127 us; error 47 us; skew 500 ppm
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:59.022053 1405 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:55:59.022483 1405 webserver.cc:460] Webserver started at http://127.20.199.3:45307/ using document root <none> and password file <none>
I20260925 14:55:59.022755 1405 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:59.022828 1405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:59.022974 1405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:59.036305 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "ec0bace055dd424f9f84a9d3d92effcc"
format_stamp: "Formatted at 2026-09-25 14:55:59 on dist-test-slave-8qt6"
server_key: "1864709af0d70777e7e0f2bfdc7a1295"
server_key_iv: "47108bef252148f3b6411c63260266f7"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.036862 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "ec0bace055dd424f9f84a9d3d92effcc"
format_stamp: "Formatted at 2026-09-25 14:55:59 on dist-test-slave-8qt6"
server_key: "1864709af0d70777e7e0f2bfdc7a1295"
server_key_iv: "47108bef252148f3b6411c63260266f7"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.038895 1405 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:55:59.061535 1420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.061810 1405 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260925 14:55:59.062004 1405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "ec0bace055dd424f9f84a9d3d92effcc"
format_stamp: "Formatted at 2026-09-25 14:55:59 on dist-test-slave-8qt6"
server_key: "1864709af0d70777e7e0f2bfdc7a1295"
server_key_iv: "47108bef252148f3b6411c63260266f7"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.062142 1405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:59.077280 1405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:59.083609 1405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:59.085659 1405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:59.086259 1405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:59.086700 1405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:59.086795 1405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.086871 1405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:59.086931 1405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.094228 1405 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:37803
I20260925 14:55:59.094810 1405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:59.097817 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:37803 every 8 connection(s)
I20260925 14:55:59.104763 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1405
I20260925 14:55:59.104858 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:59.105050 21276 external_mini_cluster.cc:1428] Setting key 324e5ab0dafd2d5dcdcad895f65038bf
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:59.123747 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.106306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:42007 (local address 127.20.199.62:46557)
0925 14:55:59.106347 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.106349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.106532 (+ 183us) server_negotiation.cc:407] Connection header received
0925 14:55:59.107203 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.107205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.107217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.107246 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.108135 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.108922 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.110665 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.110969 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.111017 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.119047 (+ 8030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:59.119048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:59.119049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:59.119057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:59.120943 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.122586 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.122587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.122588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.122649 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.122971 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.122972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.122973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.123116 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:59.123170 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:59.123556 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 14:55:59.123650 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:59.124544 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.106003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46557 (local address 127.20.199.3:42007)
0925 14:55:59.106390 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:59.106443 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:59.107076 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:59.107429 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:59.107433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:59.107735 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:59.108019 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.108027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.109215 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.109216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.110557 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.110560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.111125 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.111126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.111274 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.112013 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:59.112023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:59.118926 (+ 6903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:59.122145 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.122155 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.122158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.122489 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.122732 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.122733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.122733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.122886 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.123206 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:59.123212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:59.123385 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:55:59.124131 (+ 746us) client_negotiation.cc:240] Negotiation successful
0925 14:55:59.124300 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
I20260925 14:55:59.125217 1534 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46557
I20260925 14:55:59.125337 1534 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.125572 1534 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:55:59.125969 1050 ts_manager.cc:194] Registered new tserver with Master: ec0bace055dd424f9f84a9d3d92effcc (127.20.199.3:37803)
I20260925 14:55:59.126447 1050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:42007
I20260925 14:55:59.129236 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:59.129981 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1131
I20260925 14:55:59.136991 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1268
I20260925 14:55:59.143308 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1405
I20260925 14:55:59.153133 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1018
I20260925 14:55:59.158461 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46557
--webserver_interface=127.20.199.62
--webserver_port=45129
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:46557
--ranger_config_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:59.251946 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:59.252154 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:59.252175 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:59.253264 1537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:59.253300 1537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:59.253314 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:59.253327 1537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:59.253338 1537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:59.254963 1537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:46557
--ranger_config_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=45129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:59.255148 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:59.255322 1537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:59.260198 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.260231 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.260210 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:59.260454 1537 server_base.cc:1034] running on GCE node
I20260925 14:55:59.260725 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:59.261010 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:59.262177 1537 hybrid_clock.cc:648] HybridClock initialized: now 1790348159262161 us; error 27 us; skew 500 ppm
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:59.264667 1537 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:55:59.265133 1537 webserver.cc:460] Webserver started at http://127.20.199.62:45129/ using document root <none> and password file <none>
I20260925 14:55:59.265374 1537 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:59.265513 1537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:59.312168 1537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:59.312952 1553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.313171 1537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:59.313266 1537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "58809930c9ae4f1d9083f15cb11a373b"
format_stamp: "Formatted at 2026-09-25 14:55:57 on dist-test-slave-8qt6"
server_key: "b7fa6abaf93dcaf0a1f091cd992eb7f2"
server_key_iv: "d69350a2972ab655011afab1801ef6b1"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.313714 1537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:59.326112 1537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:59.328106 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:59.328270 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:59.332465 1537 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:46557
I20260925 14:55:59.332506 1605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:46557 every 8 connection(s)
I20260925 14:55:59.332877 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:59.334076 1606 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:59.334846 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1537
I20260925 14:55:59.334897 1606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Bootstrap starting.
I20260925 14:55:59.335151 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:43047
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:46557
--webserver_port=44341
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:59.343511 1606 log.cc:824] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:59.345264 1606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:59.345611 1606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Bootstrap complete.
I20260925 14:55:59.348004 1606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:59.348253 1606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58809930c9ae4f1d9083f15cb11a373b, State: Initialized, Role: FOLLOWER
I20260925 14:55:59.348462 1606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:59.348526 1606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:59.348589 1606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:59.348636 1606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:55:59.349747 1606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:59.349864 1606 leader_election.cc:302] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58809930c9ae4f1d9083f15cb11a373b; no voters:
I20260925 14:55:59.350090 1606 leader_election.cc:288] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:55:59.350129 1610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:55:59.350241 1610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [term 2 LEADER]: Becoming Leader. State: Replica: 58809930c9ae4f1d9083f15cb11a373b, State: Running, Role: LEADER
I20260925 14:55:59.350340 1610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } }
I20260925 14:55:59.350394 1606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:59.350627 1610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "58809930c9ae4f1d9083f15cb11a373b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } } }
I20260925 14:55:59.350698 1610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:59.350821 1610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58809930c9ae4f1d9083f15cb11a373b. Latest consensus state: current_term: 2 leader_uuid: "58809930c9ae4f1d9083f15cb11a373b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58809930c9ae4f1d9083f15cb11a373b" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 46557 } } }
I20260925 14:55:59.350889 1610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:59.351168 1614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:59.351681 1614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:59.351830 1614 catalog_manager.cc:1259] Loaded cluster ID: d7f4bd948122460abca5d6e91e9edb37
I20260925 14:55:59.351871 1614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:59.352021 1606 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:55:59.355846 1614 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:59.356416 1614 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Loaded TSK: 0
I20260925 14:55:59.356735 1614 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:55:59.451227 1608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:59.451508 1608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:59.451542 1608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:59.453279 1608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:59.453348 1608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:59.453389 1608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:55:59.455564 1608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:43047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=44341
--tserver_master_addrs=127.20.199.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:59.455793 1608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:59.456032 1608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:59.462443 1644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.462492 1643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.462467 1646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:59.463515 1608 server_base.cc:1034] running on GCE node
I20260925 14:55:59.463727 1608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:59.463974 1608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:59.465157 1608 hybrid_clock.cc:648] HybridClock initialized: now 1790348159465125 us; error 43 us; skew 500 ppm
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:59.468434 1608 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:55:59.469008 1608 webserver.cc:460] Webserver started at http://127.20.199.1:44341/ using document root <none> and password file <none>
I20260925 14:55:59.469298 1608 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:59.469425 1608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:59.496253 1608 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.001s
I20260925 14:55:59.497246 1654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.497448 1608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:59.497539 1608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "0930153463d144dcbd7523c2939298a0"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "6e0ce810db00073250c9f30f51c71276"
server_key_iv: "1bb78b2ac4ea744ba5b778847effc6a6"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.498088 1608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:59.512562 1608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:59.515093 1608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:59.515267 1608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:59.515538 1608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:59.515913 1608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:59.515961 1608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.516000 1608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:59.516026 1608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.523077 1608 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:43047
I20260925 14:55:59.523367 1767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:43047 every 8 connection(s)
I20260925 14:55:59.523569 1608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:59.526110 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1608
I20260925 14:55:59.526427 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:44835
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:46557
--webserver_port=45889
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:59.538578 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.524575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46557 (local address 127.20.199.1:59095)
0925 14:55:59.525731 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:59.525780 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:59.526501 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:59.527180 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:59.527184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:59.527434 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:59.527726 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.527734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.529530 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.529531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.530818 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.530822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.531806 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.531808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.531941 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.532587 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:59.532597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:59.534159 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:59.536527 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.536534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.536536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.536848 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.537135 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.537136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.537137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.537286 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.537838 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:59.537844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:59.538005 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 14:55:59.538134 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:55:59.538309 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":85,"threads_started":1}
I20260925 14:55:59.539589 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.524640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59095 (local address 127.20.199.62:46557)
0925 14:55:59.525633 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.525644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.525903 (+ 259us) server_negotiation.cc:407] Connection header received
0925 14:55:59.526778 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.526786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.526852 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.526972 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.527880 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.529320 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.530926 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.531566 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.531683 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.534301 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:59.534305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:59.534305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:59.534330 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:59.536413 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.536944 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.536948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.536948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.537059 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.537364 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.537365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.537366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.537721 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:59.537807 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:59.539247 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0925 14:55:59.539391 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":83,"threads_started":1}
I20260925 14:55:59.540701 1768 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46557
I20260925 14:55:59.540853 1768 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.541129 1768 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
W20260925 14:55:59.542171 1768 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:46557 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:55:59.542348 1768 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.542470 1768 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:55:59.542830 1768 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.542956 1768 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
W20260925 14:55:59.543282 1768 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:55:59.646075 1772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:59.646368 1772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:59.646417 1772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:59.648160 1772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:59.648223 1772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:59.648267 1772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:55:59.650525 1772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:44835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=45889
--tserver_master_addrs=127.20.199.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:59.650782 1772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:59.651046 1772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:59.657680 1780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:59.658665 1772 server_base.cc:1034] running on GCE node
W20260925 14:55:59.659230 1778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.659441 1777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:59.659685 1772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:59.659953 1772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:59.661139 1772 hybrid_clock.cc:648] HybridClock initialized: now 1790348159661105 us; error 45 us; skew 500 ppm
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:59.665307 1772 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:55:59.665833 1772 webserver.cc:460] Webserver started at http://127.20.199.2:45889/ using document root <none> and password file <none>
I20260925 14:55:59.666105 1772 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:59.666870 1772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:59.685883 1772 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:59.689833 1788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.690047 1772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 14:55:59.690128 1772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "971f94ae769d4af7aa4a97150243e43d"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-8qt6"
server_key: "eac6b1f10d80745b45c9f8c61402ca43"
server_key_iv: "fb22de64bf7b4ee74a39fb6bb5529795"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.690591 1772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:59.718925 1772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:59.721230 1772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:59.721379 1772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:59.721599 1772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:59.721942 1772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:59.721992 1772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.722028 1772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:59.722046 1772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.728277 1772 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44835
I20260925 14:55:59.728708 1772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:59.731221 1901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44835 every 8 connection(s)
I20260925 14:55:59.733275 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1772
I20260925 14:55:59.733606 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.199.20:53839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:37803
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:46557
--webserver_port=45307
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:59.752715 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.729509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:46815 (local address 127.20.199.62:46557)
0925 14:55:59.729558 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.729561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.741721 (+ 12160us) server_negotiation.cc:407] Connection header received
0925 14:55:59.742671 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.742673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.742704 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.742741 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.743756 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.744594 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.746202 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.746512 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.746553 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.749242 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:59.749243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:59.749244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:59.749252 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:59.750988 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.751575 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.751576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.751577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.751632 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.751914 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.751915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.751916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.752049 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:59.752090 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:59.752479 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 14:55:59.752566 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:59.753638 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.729674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46557 (local address 127.20.199.2:46815)
0925 14:55:59.741513 (+ 11839us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:59.741582 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:59.742524 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:59.742962 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:59.742968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:59.743287 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:59.743606 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.743613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.744735 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.744736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.746101 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.746103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.746656 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.746657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.746817 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.747493 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:59.747503 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:59.749131 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:59.751087 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.751094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.751097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.751490 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.751709 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.751710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.751710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.751842 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.752164 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:59.752169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:59.752330 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 14:55:59.753216 (+ 886us) client_negotiation.cc:240] Negotiation successful
0925 14:55:59.753389 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11774,"thread_start_us":114,"threads_started":1}
I20260925 14:55:59.754300 1902 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46557
I20260925 14:55:59.754391 1902 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.754565 1902 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
W20260925 14:55:59.755170 1902 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:46557 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:55:59.755232 1902 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.755285 1902 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:55:59.755581 1902 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.755643 1902 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
W20260925 14:55:59.755960 1902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:55:59.862349 1905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:59.862732 1905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:59.862833 1905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:59.864580 1905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:59.864701 1905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:59.864786 1905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:55:59.866946 1905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.199.20:53839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:37803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=45307
--tserver_master_addrs=127.20.199.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:55:59.867340 1905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:59.867677 1905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:59.873618 1910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.874011 1913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:59.875233 1911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:59.876143 1905 server_base.cc:1034] running on GCE node
I20260925 14:55:59.876446 1905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:59.876821 1905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:59.884285 1905 hybrid_clock.cc:648] HybridClock initialized: now 1790348159884250 us; error 51 us; skew 500 ppm
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:59.887614 1905 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:55:59.888144 1905 webserver.cc:460] Webserver started at http://127.20.199.3:45307/ using document root <none> and password file <none>
I20260925 14:55:59.888440 1905 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:59.888506 1905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:55:59.906268 1905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 14:55:59.914738 1922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.914991 1905 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260925 14:55:59.915083 1905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "ec0bace055dd424f9f84a9d3d92effcc"
format_stamp: "Formatted at 2026-09-25 14:55:59 on dist-test-slave-8qt6"
server_key: "1864709af0d70777e7e0f2bfdc7a1295"
server_key_iv: "47108bef252148f3b6411c63260266f7"
server_key_version: "kuduclusterkey@0"
I20260925 14:55:59.915575 1905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:59.935495 1905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:59.939510 1905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:59.939854 1905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:59.940229 1905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:59.940721 1905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:59.940814 1905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.940891 1905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:59.940951 1905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:59.948324 1905 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:37803
I20260925 14:55:59.948915 1905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:59.952234 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 1905
I20260925 14:55:59.965564 2035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:37803 every 8 connection(s)
I20260925 14:55:59.971997 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.952693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.20.199.62:46557)
0925 14:55:59.952765 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.952768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.958370 (+ 5602us) server_negotiation.cc:407] Connection header received
0925 14:55:59.958501 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.958503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.958516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.958548 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.959407 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.960183 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.961331 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.961683 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.961774 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.968171 (+ 6397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:59.968173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:59.968173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:59.968183 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:59.970689 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.971109 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.971110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.971110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.971164 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.971346 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.971347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.971347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.971470 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:59.971602 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:59.971816 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 14:55:59.971897 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:55:59 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:55:59.981948 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.966900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:48297 (local address 127.20.199.62:46557)
0925 14:55:59.972077 (+ 5177us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.972078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.972085 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:59.972102 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.972102 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.972111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.972160 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.973706 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.974454 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.975924 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.976220 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.976262 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.978732 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:59.978734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:59.978734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:59.978741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:59.980301 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.980769 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.980770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.980771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.980821 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:59.981057 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:59.981058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:59.981058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:59.981177 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:59.981224 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:59.981752 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 14:55:59.981835 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5163,"thread_start_us":55,"threads_started":1}
I20260925 14:55:59.982733 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.966594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:46557 (local address 127.20.199.3:48297)
0925 14:55:59.968854 (+ 2260us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:59.968910 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:59.969689 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:59.972965 (+ 3276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:59.972970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:59.973297 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:59.973594 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.973601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.974556 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.974556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.975819 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:59.975822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:59.976369 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:59.976370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:59.976514 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:59.977103 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:59.977114 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:59.978591 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:59.980394 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.980401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.980404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.980688 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.980881 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:59.980882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:59.980882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:59.980994 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:59.981252 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:59.981257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:59.981453 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 14:55:59.982314 (+ 861us) client_negotiation.cc:240] Negotiation successful
0925 14:55:59.982480 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2204,"thread_start_us":99,"threads_started":1}
I20260925 14:55:59.983410 2036 heartbeater.cc:347] Connected to a master server at 127.20.199.62:46557
I20260925 14:55:59.983508 2036 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.983706 2036 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
W20260925 14:55:59.984792 2036 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:46557 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:55:59.984869 2036 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.984931 2036 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:55:59.985262 2036 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.985332 2036 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
W20260925 14:55:59.985688 2036 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:56:00.543499 1768 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:00.543656 1768 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:56:00.544041 1570 ts_manager.cc:194] Registered new tserver with Master: 0930153463d144dcbd7523c2939298a0 (127.20.199.1:43047)
I20260925 14:56:00.545047 1570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:59095
Sep 25 14:56:00 dist-test-slave-8qt6 krb5kdc[574](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:00.756160 1902 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:00.756350 1902 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:56:00.756799 1570 ts_manager.cc:194] Registered new tserver with Master: 971f94ae769d4af7aa4a97150243e43d (127.20.199.2:44835)
I20260925 14:56:00.757352 1570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:46815
Sep 25 14:56:00 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:56:00.985895 2036 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:00.986078 2036 heartbeater.cc:511] Master 127.20.199.62:46557 requested a full tablet report, sending...
I20260925 14:56:00.986420 1570 ts_manager.cc:194] Registered new tserver with Master: ec0bace055dd424f9f84a9d3d92effcc (127.20.199.3:37803)
I20260925 14:56:00.986949 1570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:48297
I20260925 14:56:00.988515 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:00.997932 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.989562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.20.199.62:46557)
0925 14:56:00.989715 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.989717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.989882 (+ 165us) server_negotiation.cc:407] Connection header received
0925 14:56:00.989905 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.989906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.989919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.989979 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:00.990739 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.991493 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.992521 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.992834 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.992902 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.994508 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:00.994510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:00.994510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:00.994517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:00.996385 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.996782 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.996783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.996784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.996846 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.997049 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.997050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.997050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.997177 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:00.997288 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.997748 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.997832 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260925 14:56:01.000114 1570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:01.002499 1570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:01.026171 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.012400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44835 (local address 127.0.0.1:33680)
0925 14:56:01.015638 (+ 3238us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.015650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:01.015702 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:01.016613 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:01.016614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:01.016617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:01.016667 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.016669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.018990 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.018991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.020133 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.020136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.025851 (+ 5715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.025852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.025937 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.025951 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:56:01.026022 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:56:01.026073 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3203,"thread_start_us":72,"threads_started":1}
I20260925 14:56:01.026702 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.012125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43047 (local address 127.0.0.1:44924)
0925 14:56:01.013537 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.013580 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:01.013666 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:01.014211 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:01.014215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:01.014218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:01.014299 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.014301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.016019 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.016019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.018548 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.018551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.026373 (+ 7822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.026374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.026522 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.026527 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:01.026580 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:56:01.026619 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":77,"threads_started":1}
I20260925 14:56:01.027026 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.012992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.20.199.1:43047)
0925 14:56:01.013807 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:01.013812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:01.013823 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:56:01.013861 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:01.013863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:01.013880 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:01.013960 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:01.014403 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.015487 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.020660 (+ 5173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.021470 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.021550 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.021613 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:01.026893 (+ 5280us) server_negotiation.cc:299] Negotiation successful
0925 14:56:01.026943 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":64,"threads_started":1}
I20260925 14:56:01.027734 1701 tablet_service.cc:1467] Processing CreateTablet for tablet 859b1071641e44a69b8a35ba3b771811 (DEFAULT_TABLE table=test-table [id=c253ee1c089d4466b37f62859d120e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:01.028138 1701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859b1071641e44a69b8a35ba3b771811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:01.032241 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.013151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.20.199.2:44835)
0925 14:56:01.014094 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:01.014098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:01.015799 (+ 1701us) server_negotiation.cc:407] Connection header received
0925 14:56:01.015840 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:01.015843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:01.015859 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:01.015931 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:01.016767 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.018888 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.022923 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.023737 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.023819 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.023879 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:01.032100 (+ 8221us) server_negotiation.cc:299] Negotiation successful
0925 14:56:01.032157 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":60,"threads_started":1}
I20260925 14:56:01.032867 1836 tablet_service.cc:1467] Processing CreateTablet for tablet 859b1071641e44a69b8a35ba3b771811 (DEFAULT_TABLE table=test-table [id=c253ee1c089d4466b37f62859d120e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:01.033206 1836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859b1071641e44a69b8a35ba3b771811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:01.037492 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.012593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:37803 (local address 127.0.0.1:57860)
0925 14:56:01.020268 (+ 7675us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.020279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:01.020328 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:01.021665 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:01.021666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:01.021668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:01.021709 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.021711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.023919 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.023920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.024852 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.024855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.037178 (+ 12323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.037180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.037265 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.037271 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:56:01.037336 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:56:01.037390 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7656,"thread_start_us":82,"threads_started":1}
I20260925 14:56:01.037848 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.013303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.20.199.3:37803)
0925 14:56:01.014163 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:01.014167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:01.020393 (+ 6226us) server_negotiation.cc:407] Connection header received
0925 14:56:01.020434 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:01.020437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:01.020454 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:01.020526 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:01.021804 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.022803 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.024963 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.025688 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.025764 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.025812 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:01.037697 (+ 11885us) server_negotiation.cc:299] Negotiation successful
0925 14:56:01.037764 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":66,"threads_started":1}
I20260925 14:56:01.038517 1966 tablet_service.cc:1467] Processing CreateTablet for tablet 859b1071641e44a69b8a35ba3b771811 (DEFAULT_TABLE table=test-table [id=c253ee1c089d4466b37f62859d120e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:01.038895 1966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859b1071641e44a69b8a35ba3b771811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:01.042986 2071 tablet_bootstrap.cc:492] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Bootstrap starting.
I20260925 14:56:01.043697 2071 tablet_bootstrap.cc:654] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:01.044050 2071 log.cc:824] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:01.044430 2070 tablet_bootstrap.cc:492] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Bootstrap starting.
I20260925 14:56:01.045113 2070 tablet_bootstrap.cc:654] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:01.045585 2069 tablet_bootstrap.cc:492] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: Bootstrap starting.
I20260925 14:56:01.046254 2069 tablet_bootstrap.cc:654] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:01.046588 2069 log.cc:824] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:01.046927 2070 log.cc:824] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:01.047783 2071 tablet_bootstrap.cc:492] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: No bootstrap required, opened a new log
I20260925 14:56:01.047868 2071 ts_tablet_manager.cc:1397] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:56:01.050585 2071 raft_consensus.cc:348] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.050796 2071 raft_consensus.cc:374] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:01.050841 2071 raft_consensus.cc:729] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0bace055dd424f9f84a9d3d92effcc, State: Initialized, Role: FOLLOWER
I20260925 14:56:01.050988 2071 consensus_queue.cc:260] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.051268 2071 ts_tablet_manager.cc:1428] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:56:01.051397 2036 heartbeater.cc:503] Master 127.20.199.62:46557 was elected leader, sending a full tablet report...
I20260925 14:56:01.052991 2069 tablet_bootstrap.cc:492] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: No bootstrap required, opened a new log
I20260925 14:56:01.053078 2069 ts_tablet_manager.cc:1397] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 14:56:01.055565 2069 raft_consensus.cc:348] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.055786 2069 raft_consensus.cc:374] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:01.055848 2069 raft_consensus.cc:729] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0930153463d144dcbd7523c2939298a0, State: Initialized, Role: FOLLOWER
I20260925 14:56:01.055998 2069 consensus_queue.cc:260] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.056617 2069 ts_tablet_manager.cc:1428] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:56:01.056787 1768 heartbeater.cc:503] Master 127.20.199.62:46557 was elected leader, sending a full tablet report...
I20260925 14:56:01.058305 2070 tablet_bootstrap.cc:492] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: No bootstrap required, opened a new log
I20260925 14:56:01.058405 2070 ts_tablet_manager.cc:1397] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20260925 14:56:01.061450 2070 raft_consensus.cc:348] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.061661 2070 raft_consensus.cc:374] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:01.061717 2070 raft_consensus.cc:729] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f94ae769d4af7aa4a97150243e43d, State: Initialized, Role: FOLLOWER
I20260925 14:56:01.061896 2070 consensus_queue.cc:260] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.062202 2070 ts_tablet_manager.cc:1428] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260925 14:56:01.062352 1902 heartbeater.cc:503] Master 127.20.199.62:46557 was elected leader, sending a full tablet report...
Sep 25 14:56:01 dist-test-slave-8qt6 krb5kdc[574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for HTTP/127.20.199.20@KRBTEST.COM
I20260925 14:56:01.216320 2077 raft_consensus.cc:482] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:01.216480 2077 raft_consensus.cc:504] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.216892 2077 leader_election.cc:288] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0930153463d144dcbd7523c2939298a0 (127.20.199.1:43047), ec0bace055dd424f9f84a9d3d92effcc (127.20.199.3:37803)
W20260925 14:56:01.217175 2037 tablet.cc:2369] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:01.232355 1903 tablet.cc:2369] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:01.234843 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.217376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:55485 (local address 127.20.199.1:43047)
0925 14:56:01.217554 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:01.217557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:01.225527 (+ 7970us) server_negotiation.cc:407] Connection header received
0925 14:56:01.230850 (+ 5323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:01.230852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:01.230865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:01.230903 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:01.231202 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.231986 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.233856 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.234388 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.234444 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.234456 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:01.234700 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:56:01.234749 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 14:56:01.235162 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.217091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43047 (local address 127.20.199.2:55485)
0925 14:56:01.225408 (+ 8317us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.225425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:01.230689 (+ 5264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:01.230995 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:01.230996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:01.230998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:01.231101 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.231104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.232461 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.232462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.233714 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.233719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.234547 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.234549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.234615 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.234619 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:01.235013 (+ 394us) client_negotiation.cc:240] Negotiation successful
0925 14:56:01.235062 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8283,"thread_start_us":113,"threads_started":1}
I20260925 14:56:01.235569 1722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859b1071641e44a69b8a35ba3b771811" candidate_uuid: "971f94ae769d4af7aa4a97150243e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0930153463d144dcbd7523c2939298a0" is_pre_election: true
I20260925 14:56:01.235762 1722 raft_consensus.cc:2454] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f94ae769d4af7aa4a97150243e43d in term 0.
I20260925 14:56:01.236029 1791 leader_election.cc:302] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0930153463d144dcbd7523c2939298a0, 971f94ae769d4af7aa4a97150243e43d; no voters:
I20260925 14:56:01.236189 2077 raft_consensus.cc:2781] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:01.236243 2077 raft_consensus.cc:482] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:01.236266 2077 raft_consensus.cc:3037] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:01.237299 2077 raft_consensus.cc:504] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.237480 2077 leader_election.cc:288] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [CANDIDATE]: Term 1 election: Requested vote from peers 0930153463d144dcbd7523c2939298a0 (127.20.199.1:43047), ec0bace055dd424f9f84a9d3d92effcc (127.20.199.3:37803)
I20260925 14:56:01.237664 1722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859b1071641e44a69b8a35ba3b771811" candidate_uuid: "971f94ae769d4af7aa4a97150243e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0930153463d144dcbd7523c2939298a0"
I20260925 14:56:01.237757 1722 raft_consensus.cc:3037] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:01.238672 1722 raft_consensus.cc:2454] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f94ae769d4af7aa4a97150243e43d in term 1.
I20260925 14:56:01.238876 1791 leader_election.cc:302] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0930153463d144dcbd7523c2939298a0, 971f94ae769d4af7aa4a97150243e43d; no voters:
I20260925 14:56:01.238965 2077 raft_consensus.cc:2781] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:01.239127 2077 raft_consensus.cc:686] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 1 LEADER]: Becoming Leader. State: Replica: 971f94ae769d4af7aa4a97150243e43d, State: Running, Role: LEADER
I20260925 14:56:01.239238 2077 consensus_queue.cc:237] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } }
I20260925 14:56:01.240126 1569 catalog_manager.cc:5495] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 971f94ae769d4af7aa4a97150243e43d (127.20.199.2). New cstate: current_term: 1 leader_uuid: "971f94ae769d4af7aa4a97150243e43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971f94ae769d4af7aa4a97150243e43d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:01.244431 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.217618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:53403 (local address 127.20.199.3:37803)
0925 14:56:01.217661 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:01.217663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:01.235515 (+ 17852us) server_negotiation.cc:407] Connection header received
0925 14:56:01.235649 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:01.235650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:01.235661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:01.235693 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:01.240589 (+ 4896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.241290 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.243376 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.243979 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.244034 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.244044 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:01.244301 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 14:56:01.244354 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:56:01.244720 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.217198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:37803 (local address 127.20.199.2:53403)
0925 14:56:01.235442 (+ 18244us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.235456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:01.235546 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:01.235759 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:01.235760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:01.235762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:01.235830 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.235831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.241472 (+ 5641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.241473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.242681 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:01.242686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.244111 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.244112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:01.244192 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.244214 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:56:01.244587 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 14:56:01.244635 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18221,"thread_start_us":131,"threads_started":1}
I20260925 14:56:01.245131 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859b1071641e44a69b8a35ba3b771811" candidate_uuid: "971f94ae769d4af7aa4a97150243e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0bace055dd424f9f84a9d3d92effcc" is_pre_election: true
I20260925 14:56:01.245306 1990 raft_consensus.cc:2454] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f94ae769d4af7aa4a97150243e43d in term 0.
I20260925 14:56:01.245465 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859b1071641e44a69b8a35ba3b771811" candidate_uuid: "971f94ae769d4af7aa4a97150243e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0bace055dd424f9f84a9d3d92effcc"
I20260925 14:56:01.245545 1989 raft_consensus.cc:3037] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:01.246932 1989 raft_consensus.cc:2454] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f94ae769d4af7aa4a97150243e43d in term 1.
W20260925 14:56:01.276698 1769 tablet.cc:2369] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:01.299592 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.296125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.20.199.2:44835)
0925 14:56:01.296169 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:01.296171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:01.296200 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:56:01.296245 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:01.296246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:01.296270 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:01.296342 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:01.296531 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.297291 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.298452 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:01.298735 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:01.298807 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:01.299018 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:01.299159 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:01.299245 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:01.299481 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:56:01.299505 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:56:01.302294 1989 raft_consensus.cc:1261] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 1 FOLLOWER]: Refusing update from remote peer 971f94ae769d4af7aa4a97150243e43d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:01.302816 1722 raft_consensus.cc:1261] T 859b1071641e44a69b8a35ba3b771811 P 0930153463d144dcbd7523c2939298a0 [term 1 FOLLOWER]: Refusing update from remote peer 971f94ae769d4af7aa4a97150243e43d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:01.303260 2077 consensus_queue.cc:1035] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0bace055dd424f9f84a9d3d92effcc" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:01.303751 2077 consensus_queue.cc:1035] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0930153463d144dcbd7523c2939298a0" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:01.311473 2085 mvcc.cc:205] Tried to move back new op lower bound from 7333266068690354176 to 7333266068436668416. Current Snapshot: MvccSnapshot[applied={T|T < 7333266068690354176}]
I20260925 14:56:01.311686 2086 mvcc.cc:205] Tried to move back new op lower bound from 7333266068690354176 to 7333266068436668416. Current Snapshot: MvccSnapshot[applied={T|T < 7333266068690354176}]
I20260925 14:56:01.323486 2088 mvcc.cc:205] Tried to move back new op lower bound from 7333266068690354176 to 7333266068436668416. Current Snapshot: MvccSnapshot[applied={T|T < 7333266068690354176}]
I20260925 14:56:01.327867 1570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:01.327993 1570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:01.328704 1570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58809930c9ae4f1d9083f15cb11a373b: Sending DeleteTablet for 3 replicas of tablet 859b1071641e44a69b8a35ba3b771811
I20260925 14:56:01.329218 1701 tablet_service.cc:1514] Processing DeleteTablet for tablet 859b1071641e44a69b8a35ba3b771811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:01 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:44924
I20260925 14:56:01.329528 1966 tablet_service.cc:1514] Processing DeleteTablet for tablet 859b1071641e44a69b8a35ba3b771811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:01 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:57860
I20260925 14:56:01.329878 1836 tablet_service.cc:1514] Processing DeleteTablet for tablet 859b1071641e44a69b8a35ba3b771811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:01 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:33680
I20260925 14:56:01.330435 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1608
W20260925 14:56:01.337831 1556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:43047 (error 108)
I20260925 14:56:01.338179 2095 tablet_replica.cc:330] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: stopping tablet replica
I20260925 14:56:01.338328 2095 raft_consensus.cc:2229] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:01.338457 2095 raft_consensus.cc:2258] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:01.338629 2096 tablet_replica.cc:330] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: stopping tablet replica
I20260925 14:56:01.338737 2096 raft_consensus.cc:2229] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:01.338871 2096 raft_consensus.cc:2258] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:01.339797 2095 ts_tablet_manager.cc:1905] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:01.340054 2096 ts_tablet_manager.cc:1905] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:01.342233 2096 ts_tablet_manager.cc:1918] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:01.342312 2096 log.cc:1196] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/859b1071641e44a69b8a35ba3b771811
I20260925 14:56:01.342603 2096 ts_tablet_manager.cc:1939] T 859b1071641e44a69b8a35ba3b771811 P 971f94ae769d4af7aa4a97150243e43d: Deleting consensus metadata
I20260925 14:56:01.343816 2095 ts_tablet_manager.cc:1918] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:01.345477 2095 log.cc:1196] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/859b1071641e44a69b8a35ba3b771811
I20260925 14:56:01.345795 1556 catalog_manager.cc:4841] TS 971f94ae769d4af7aa4a97150243e43d (127.20.199.2:44835): tablet 859b1071641e44a69b8a35ba3b771811 (table test-table [id=c253ee1c089d4466b37f62859d120e6d]) successfully deleted
I20260925 14:56:01.345932 2095 ts_tablet_manager.cc:1939] T 859b1071641e44a69b8a35ba3b771811 P ec0bace055dd424f9f84a9d3d92effcc: Deleting consensus metadata
W20260925 14:56:01.346086 2063 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:56:01.341731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43047 (local address 127.0.0.1:44926)
0925 14:56:01.341791 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.341803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:01.341847 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:01.346038 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:56:01.346220 1556 catalog_manager.cc:4584] TS 0930153463d144dcbd7523c2939298a0 (127.20.199.1:43047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 859b1071641e44a69b8a35ba3b771811: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:56:01.346597 1555 catalog_manager.cc:4841] TS ec0bace055dd424f9f84a9d3d92effcc (127.20.199.3:37803): tablet 859b1071641e44a69b8a35ba3b771811 (table test-table [id=c253ee1c089d4466b37f62859d120e6d]) successfully deleted
I20260925 14:56:01.350737 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1772
I20260925 14:56:01.366045 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1905
I20260925 14:56:01.371771 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 1537
I20260925 14:56:01.379163 21276 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:56:01.723840 21276 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:56:01.723933 21276 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:56:02.068583 21276 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:56:02.068 UTC [604] LOG: received smart shutdown request
2026-09-25 14:56:02.072 UTC [604] LOG: background worker "logical replication launcher" (PID 613) exited with exit code 1
2026-09-25 14:56:02.072 UTC [608] LOG: shutting down
2026-09-25 14:56:02.247 UTC [604] LOG: database system is shut down
2026-09-25T14:56:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46932 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:02 dist-test-slave-8qt6 krb5kdc[2106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:02 dist-test-slave-8qt6 krb5kdc[2106](info): set up 2 sockets
Sep 25 14:56:02 dist-test-slave-8qt6 krb5kdc[2106](info): commencing operation
krb5kdc: starting...
W20260925 14:56:03.393536 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:56:03.516044 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:39759
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:39759,127.20.199.61:37381,127.20.199.60:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:03.596779 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:03.597026 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:03.597050 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:03.598268 2119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:03.598311 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:03.598325 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:03.598338 2119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:03.598395 2119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:03.599922 2119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:39759,127.20.199.61:37381,127.20.199.60:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:03.600172 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:03.600423 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:03.605329 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:03.605356 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:03.605357 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:03.605584 2119 server_base.cc:1034] running on GCE node
I20260925 14:56:03.605840 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:03.606146 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:03.607374 2119 hybrid_clock.cc:648] HybridClock initialized: now 1790348163607344 us; error 37 us; skew 500 ppm
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:03.614904 2119 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:03.615372 2119 webserver.cc:460] Webserver started at http://127.20.199.62:39889/ using document root <none> and password file <none>
I20260925 14:56:03.615579 2119 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:03.615633 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:03.615733 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:03.616820 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "fb5069f1368948da93e6706bc83bf08f"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "52f8716947f4811287a2483dc82bad28"
server_key_iv: "19ac3ef687273f1a73a74db475dd314d"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.617211 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "fb5069f1368948da93e6706bc83bf08f"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "52f8716947f4811287a2483dc82bad28"
server_key_iv: "19ac3ef687273f1a73a74db475dd314d"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.618750 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:56:03.619573 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:03.619859 2119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:03.619967 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "fb5069f1368948da93e6706bc83bf08f"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "52f8716947f4811287a2483dc82bad28"
server_key_iv: "19ac3ef687273f1a73a74db475dd314d"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.620046 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:03.629030 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:03.631255 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:03.631429 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:03.635527 2119 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:39759
I20260925 14:56:03.635557 2186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:39759 every 8 connection(s)
I20260925 14:56:03.635927 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:03.636811 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:03.638628 2187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.640719 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2119
I20260925 14:56:03.640805 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:56:03.640993 21276 external_mini_cluster.cc:1428] Setting key 78d25b436ddeab38ad886217e2018702
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.61@KRBTEST.COM; defaulting to no policy
I20260925 14:56:03.650162 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.639383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:39759 (local address 127.0.0.1:53510)
0925 14:56:03.639568 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.639613 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.640115 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.640414 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.640423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.640739 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.640976 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.640982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.642640 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.642642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.643872 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.643875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.644376 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.644377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.644457 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.645005 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.645017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.646708 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.648552 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.648558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.648560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.648750 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.649014 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.649014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.649015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.649129 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.649548 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.649553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.649622 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.649726 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.649975 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":100,"thread_start_us":104,"threads_started":1}
I20260925 14:56:03.650166 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.639473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.20.199.62:39759)
0925 14:56:03.639622 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.639629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.639726 (+ 97us) server_negotiation.cc:407] Connection header received
0925 14:56:03.640252 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.640254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.640271 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.640370 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.641113 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.642491 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.643959 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.644276 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.644362 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.646819 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.646826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.646826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.646839 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.648450 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.648851 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.648854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.648854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.648935 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.649187 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.649188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.649188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.649470 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.649557 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.649751 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.649911 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260925 14:56:03.650997 2187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.651382 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.651260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40782)
0925 14:56:03.651305 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.651361 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:56:03.651556 2136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.61@KRBTEST.COM" created.
I20260925 14:56:03.651998 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.651901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40784)
0925 14:56:03.651949 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.651974 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:56:03.652154 2187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:37381: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab.
Entry for principal kudu/127.20.199.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab.
I20260925 14:56:03.666471 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:37381
--webserver_interface=127.20.199.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab
--principal=kudu/127.20.199.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:39759,127.20.199.61:37381,127.20.199.60:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:03.701429 2187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } attempt: 1
I20260925 14:56:03.701992 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.701810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40800)
0925 14:56:03.701890 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.701946 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:56:03.702406 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.702279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40804)
0925 14:56:03.702333 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.702371 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:56:03.702549 2187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:37381: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
W20260925 14:56:03.749099 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:03.749328 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:03.749352 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:03.750538 2194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:03.750586 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:03.750600 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:03.750612 2194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:03.750624 2194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:03.752203 2194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:39759,127.20.199.61:37381,127.20.199.60:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab
--principal=kudu/127.20.199.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.199.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:03.752470 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:03.752722 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:03.757902 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:03.757896 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:03.758072 2194 server_base.cc:1034] running on GCE node
W20260925 14:56:03.757896 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:03.758440 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:03.758826 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:03.760005 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790348163759983 us; error 36 us; skew 500 ppm
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:03.762673 2194 init.cc:377] Logged in from keytab as kudu/127.20.199.61@KRBTEST.COM (short username kudu)
I20260925 14:56:03.763478 2194 webserver.cc:460] Webserver started at http://127.20.199.61:44335/ using document root <none> and password file <none>
I20260925 14:56:03.763758 2194 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:03.763836 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:03.763993 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:03.765112 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data/instance:
uuid: "d6c3267555f2436388d73505f18d8372"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "65cb67836f110f5bb4c167951e8e3686"
server_key_iv: "fa2aa5ba567f884ef036a95bb05d7dd3"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.765552 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal/instance:
uuid: "d6c3267555f2436388d73505f18d8372"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "65cb67836f110f5bb4c167951e8e3686"
server_key_iv: "fa2aa5ba567f884ef036a95bb05d7dd3"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.766893 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:03.767637 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:03.767827 2194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:56:03.767938 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal
uuid: "d6c3267555f2436388d73505f18d8372"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "65cb67836f110f5bb4c167951e8e3686"
server_key_iv: "fa2aa5ba567f884ef036a95bb05d7dd3"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.768028 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:03.770723 2187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } attempt: 2
I20260925 14:56:03.771242 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.771079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40820)
0925 14:56:03.771152 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.771198 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:56:03.771632 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.771515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40824)
0925 14:56:03.771571 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.771602 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:56:03.771752 2187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:37381: Network error: Client connection negotiation failed: client connection to 127.20.199.61:37381: connect: Connection refused (error 111)
I20260925 14:56:03.788715 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:03.791165 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:03.791283 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:03.795430 2194 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.61:37381
I20260925 14:56:03.795487 2261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.61:37381 every 8 connection(s)
I20260925 14:56:03.795835 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
I20260925 14:56:03.796684 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:03.798229 2262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.801339 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2194
I20260925 14:56:03.801522 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-1/wal/instance
I20260925 14:56:03.801761 21276 external_mini_cluster.cc:1428] Setting key 4fe14da9453b25719eeb4dbf34a41cac
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:03.808208 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.798953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.20.199.62:39759)
0925 14:56:03.799011 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.799014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.799204 (+ 190us) server_negotiation.cc:407] Connection header received
0925 14:56:03.799663 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.799664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.799675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.799734 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.800341 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.800829 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.802064 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.802403 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.802485 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.805204 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.805205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.805205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.805213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.806702 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.807246 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.807246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.807247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.807296 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.807581 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.807581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.807582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.807680 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.807730 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.808079 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.808142 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:56:03.808501 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.798891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:39759 (local address 127.0.0.1:53522)
0925 14:56:03.799069 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.799113 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.799569 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.799790 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.799793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.800054 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.800261 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.800265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.800938 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.800939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.801977 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.801980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.802520 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.802522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.802669 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.803322 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.803330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.805091 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.806790 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.806799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.806802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.807164 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.807366 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.807367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.807368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.807513 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.807749 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.807757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.807938 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.808064 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.808247 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260925 14:56:03.809140 2262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.199.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.60@KRBTEST.COM" created.
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:56:03.817320 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.809473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40832)
0925 14:56:03.809506 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.809517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.809559 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.809841 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.809841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.809950 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.809999 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.810001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.810972 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.810973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.811466 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.811467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.811918 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.811919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.811969 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.812301 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.812304 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.813726 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.816191 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.816192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.816192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.816355 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.816595 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.816596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.816596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.816648 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.817046 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.817047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.817102 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.817174 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.817252 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 14:56:03.817526 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.809516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.20.199.61:37381)
0925 14:56:03.809681 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.809686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.809708 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:56:03.809730 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.809732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.809748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.809841 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.810076 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.810899 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.811530 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.811822 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.811933 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.813842 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.813850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.813850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.813857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.816080 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.816443 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.816447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.816447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.816521 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.816716 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.816717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.816718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.816947 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.817057 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.817245 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.817344 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260925 14:56:03.818267 2262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.818717 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.818543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33714)
0925 14:56:03.818604 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.818678 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:56:03.818890 2213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
I20260925 14:56:03.819414 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.819280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33730)
0925 14:56:03.819342 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.819380 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:56:03.819546 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:36635: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab.
Entry for principal kudu/127.20.199.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab.
I20260925 14:56:03.832006 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:36635
--webserver_interface=127.20.199.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab
--principal=kudu/127.20.199.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:39759,127.20.199.61:37381,127.20.199.60:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:03.862924 2187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } attempt: 3
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:56:03.868803 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } attempt: 1
I20260925 14:56:03.869208 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.869060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33746)
0925 14:56:03.869126 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.869168 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:56:03.869707 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.869479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33760)
0925 14:56:03.869638 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.869668 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
W20260925 14:56:03.869863 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:36635: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
I20260925 14:56:03.870306 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.863379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.20.199.61:37381)
0925 14:56:03.863448 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.863450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.863461 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:56:03.863534 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.863535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.863547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.863623 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.863947 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.864469 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.865179 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.865461 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.865525 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.867578 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.867580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.867580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.867588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.869073 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.869461 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.869462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.869462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.869525 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.869731 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.869732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.869732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.869896 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.869994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.870196 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.870257 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:56:03.870440 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.863303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40844)
0925 14:56:03.863375 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.863394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.863466 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.863629 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.863630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.863794 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.863858 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.863860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.864556 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.864557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.865079 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.865081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.865520 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.865521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.865600 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.866067 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.866079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.867490 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.869138 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.869139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.869140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.869370 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.869597 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.869598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.869598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.869657 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.870000 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.870002 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.870067 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.870180 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.870307 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:56:03.870836 2187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.871197 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.871086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33764)
0925 14:56:03.871140 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.871172 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:56:03.871523 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.871434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33768)
0925 14:56:03.871478 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.871504 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:56:03.871626 2187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:36635: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
I20260925 14:56:03.902786 2187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } attempt: 1
I20260925 14:56:03.903292 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.903134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33770)
0925 14:56:03.903206 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.903250 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:56:03.903687 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.903580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33772)
0925 14:56:03.903629 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.903661 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:56:03.903788 2187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:36635: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
W20260925 14:56:03.916141 2269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:03.916378 2269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:03.916402 2269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:03.917665 2269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:03.917716 2269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:03.917730 2269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:03.917743 2269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:03.917757 2269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:03.919359 2269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:39759,127.20.199.61:37381,127.20.199.60:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab
--principal=kudu/127.20.199.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.199.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:03.919641 2269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:03.919894 2269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:03.925035 2276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:03.924976 2278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:03.924976 2275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:03.925864 2269 server_base.cc:1034] running on GCE node
I20260925 14:56:03.926142 2269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:03.926467 2269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:03.927626 2269 hybrid_clock.cc:648] HybridClock initialized: now 1790348163927612 us; error 29 us; skew 500 ppm
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:03.930471 2269 init.cc:377] Logged in from keytab as kudu/127.20.199.60@KRBTEST.COM (short username kudu)
I20260925 14:56:03.930996 2269 webserver.cc:460] Webserver started at http://127.20.199.60:37683/ using document root <none> and password file <none>
I20260925 14:56:03.931196 2269 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:03.931238 2269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:03.931336 2269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:03.932512 2269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data/instance:
uuid: "907de4ec7b3b40a7bd94a0b362c754ef"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "a8bdbd9f9410676a7c7d96ab9ac28d54"
server_key_iv: "c059a5c1ba5bd35bb6407aea6440e8ce"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.932994 2269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal/instance:
uuid: "907de4ec7b3b40a7bd94a0b362c754ef"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "a8bdbd9f9410676a7c7d96ab9ac28d54"
server_key_iv: "c059a5c1ba5bd35bb6407aea6440e8ce"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.934928 2269 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 14:56:03.935775 2285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:03.935953 2269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:03.936062 2269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal
uuid: "907de4ec7b3b40a7bd94a0b362c754ef"
format_stamp: "Formatted at 2026-09-25 14:56:03 on dist-test-slave-8qt6"
server_key: "a8bdbd9f9410676a7c7d96ab9ac28d54"
server_key_iv: "c059a5c1ba5bd35bb6407aea6440e8ce"
server_key_version: "encryptionkey@0"
I20260925 14:56:03.936167 2269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:03.938020 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } attempt: 2
I20260925 14:56:03.938477 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.938295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33784)
0925 14:56:03.938382 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.938442 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 14:56:03.938884 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.938756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33788)
0925 14:56:03.938821 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.938855 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 14:56:03.939657 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:36635: Network error: Client connection negotiation failed: client connection to 127.20.199.60:36635: connect: Connection refused (error 111)
I20260925 14:56:03.965070 2269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:03.967522 2269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:03.967736 2269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:03.972324 2269 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.60:36635
I20260925 14:56:03.972361 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.60:36635 every 8 connection(s)
I20260925 14:56:03.972761 2269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
I20260925 14:56:03.973766 2338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:03.975325 2338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.977016 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2269
I20260925 14:56:03.977128 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348014035135-21276-0/minicluster-data/master-2/wal/instance
I20260925 14:56:03.977324 21276 external_mini_cluster.cc:1428] Setting key 829797b5be3a4d405657bc81b0e8a77e
I20260925 14:56:03.978958 2187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } attempt: 2
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:56:03.987398 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.976124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.20.199.62:39759)
0925 14:56:03.976200 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.976203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.976432 (+ 229us) server_negotiation.cc:407] Connection header received
0925 14:56:03.976931 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.976932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.976946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.977023 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.977992 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.978802 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.980527 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.980821 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.980894 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.983636 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.983638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.983638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.983645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.985380 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.986103 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.986105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.986105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.986172 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.986450 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.986451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.986451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.986653 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.986746 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.987178 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.987286 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:56:03.987947 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.976096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:39759 (local address 127.0.0.1:53538)
0925 14:56:03.976302 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.976340 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.976827 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.977091 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.977095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.977527 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.977841 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.977847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.978916 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.978918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.980447 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.980451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.980920 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.980921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.981063 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.981727 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.981737 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.983543 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.985503 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.985511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.985514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.985996 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.986252 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.986253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.986253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.986376 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.986746 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.986751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.987002 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.987154 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.987336 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260925 14:56:03.988247 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.979240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33798)
0925 14:56:03.979294 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.979304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.979343 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.979562 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.979563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.979709 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.979741 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.979742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.980460 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.980461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.981086 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.981088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.981753 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.981754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.981815 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.982270 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.982275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.984584 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.986716 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.986717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.986717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.987073 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.987327 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.987329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.987329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.987395 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.987892 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.987893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.987938 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.988007 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.988133 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:56:03.988766 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.979282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.20.199.60:36635)
0925 14:56:03.979425 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.979427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.979434 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:56:03.979459 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.979459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.979467 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.979541 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.979814 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.980363 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.981185 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.981641 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.981973 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.984682 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.984690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.984691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.984740 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.986643 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.987183 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.987186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.987186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.987247 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.987600 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.987601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.987601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.987793 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.987909 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.988547 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.988655 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260925 14:56:03.988818 2338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:03.988956 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.977765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.20.199.60:36635)
0925 14:56:03.977947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.977951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.978020 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:56:03.978087 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.978089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.978106 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.978163 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.978935 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.980178 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.981172 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.981630 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.981707 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.985249 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.985250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.985251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.985257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.987670 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.988119 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.988121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.988121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.988175 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.988405 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.988406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.988406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.988534 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.988688 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.988780 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.988858 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
I20260925 14:56:03.990978 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: Bootstrap starting.
I20260925 14:56:03.991900 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:03.992229 2187 log.cc:824] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:03.993693 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: No bootstrap required, opened a new log
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:03.995793 2187 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:03.995965 2187 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:03.996001 2187 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5069f1368948da93e6706bc83bf08f, State: Initialized, Role: FOLLOWER
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:56:03.996140 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:03.996487 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:03.996753 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:03.997195 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:03.999595 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.989072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:37381 (local address 127.0.0.1:40850)
0925 14:56:03.989329 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:03.989340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:03.989449 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:03.992501 (+ 3052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:03.992502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:03.992619 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:03.992708 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.992710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.993307 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.993308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.994240 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:03.994243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.994669 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.994670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:03.994731 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.995117 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:03.995122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:03.996452 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:03.998431 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.998432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.998433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.998662 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.998906 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:03.998907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:03.998907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:03.998967 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:03.999272 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:03.999273 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:03.999319 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:56:03.999381 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:56:03.999487 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260925 14:56:03.999629 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.990040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.20.199.62:39759)
0925 14:56:03.990186 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.990187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.990193 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:56:03.990208 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.990209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.990216 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.990281 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.990773 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.991521 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.992349 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.992617 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.992701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.995497 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.995498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.995499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.995507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.997992 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.998462 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.998463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.998464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.998515 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.998801 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.998802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.998803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.998939 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.999419 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.999464 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.999545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 14:56:03.999908 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:03.989138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.20.199.61:37381)
0925 14:56:03.989284 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:03.989285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:03.989435 (+ 150us) server_negotiation.cc:407] Connection header received
0925 14:56:03.989509 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:03.989510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:03.989523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:03.989578 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:03.992793 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.993244 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.994359 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:03.994572 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:03.994606 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:03.996546 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:03.996548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:03.996548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:03.996557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:03.998337 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.998744 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.998745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.998746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.998825 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:03.999032 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:03.999033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:03.999034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:03.999170 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:03.999243 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:03.999731 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 14:56:03.999830 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 14:56:04.000217 2338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:56:04.000558 2119 master_runner.cc:186] Error getting master registration for 127.20.199.61:37381: OK, instance_id { permanent_uuid: "d6c3267555f2436388d73505f18d8372" instance_seqno: 1790348163794205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:56:04.004391 2119 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 14:56:03.998327 2356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:56:04.004593 2356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
Sep 25 14:56:03 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:56:04.009207 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.000169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.20.199.61:37381)
0925 14:56:04.000403 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:04.000406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:04.000413 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:56:04.000433 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:04.000434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:04.000445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:04.000493 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:04.001109 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.001988 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.002930 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.003302 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.003396 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.005934 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:04.005935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:04.005936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:04.005943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:04.007901 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.008322 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.008323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.008324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.008388 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.008617 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.008618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.008618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.008778 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:04.008978 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:04.009015 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:56:04.009090 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260925 14:56:04.009843 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.000487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33810)
0925 14:56:04.000527 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:04.000538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:04.000575 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:04.000698 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:04.000698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:04.000844 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:04.000885 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:04.000886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.001816 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.001817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:04.002561 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:04.002564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.003032 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.003033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:04.003083 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.003629 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:04.003634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:04.005180 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:04.008677 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:04.008678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:04.008678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:04.008892 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:04.009089 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:04.009089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:04.009089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:04.009137 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:04.009501 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:04.009502 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:04.009553 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:56:04.009630 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:56:04.009743 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 14:56:04.010308 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.000569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.20.199.60:36635)
0925 14:56:04.000605 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:04.000606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:04.000613 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:56:04.000632 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:04.000633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:04.000641 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:04.000695 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:04.000972 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.001724 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.002661 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.002952 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.003260 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.005280 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:04.005282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:04.005282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:04.005289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:04.008561 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.008975 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.008976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.008976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.009022 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.009211 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.009212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.009212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.009434 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:04.009506 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:04.009671 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 14:56:04.009871 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":20}
I20260925 14:56:04.010963 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:56:04.011969 2338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef: Bootstrap starting.
I20260925 14:56:04.012707 2338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:04.012943 2338 log.cc:824] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:04.013820 2338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef: No bootstrap required, opened a new log
I20260925 14:56:04.015954 2338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.016091 2338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:04.016145 2338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907de4ec7b3b40a7bd94a0b362c754ef, State: Initialized, Role: FOLLOWER
I20260925 14:56:04.016491 2338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.016901 2338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:56:04.019383 2377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:56:04.019507 2377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:56:04.019899 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.020061 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:04.020515 2269 master_runner.cc:186] Error getting master registration for 127.20.199.61:37381: OK, instance_id { permanent_uuid: "d6c3267555f2436388d73505f18d8372" instance_seqno: 1790348163794205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:56:04.022948 2269 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:56:04.023311 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.014014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.20.199.61:37381)
0925 14:56:04.014068 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:04.014070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:04.014450 (+ 380us) server_negotiation.cc:407] Connection header received
0925 14:56:04.014483 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:04.014484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:04.014496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:04.014556 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:04.015319 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.016064 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.017081 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.017392 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.017494 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.019870 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:04.019872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:04.019872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:04.019882 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:04.021896 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.022339 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.022340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.022341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.022412 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.022673 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.022674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.022675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.022824 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:04.022902 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:04.023115 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 14:56:04.023208 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:56:04.027051 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.016307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.20.199.62:39759)
0925 14:56:04.016381 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:04.016383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:04.016392 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:56:04.016416 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:04.016417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:04.016428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:04.016480 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:04.017217 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.017981 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.019555 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.019834 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.019899 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.023921 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:04.023923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:04.023923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:04.023931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:04.025745 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.026118 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.026119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.026120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.026185 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.026367 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.026368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.026369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.026492 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:04.026572 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:04.026871 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 14:56:04.026953 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:56:04.028616 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.014162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.20.199.60:36635)
0925 14:56:04.014203 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:04.014205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:04.020319 (+ 6114us) server_negotiation.cc:407] Connection header received
0925 14:56:04.020378 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:04.020380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:04.020393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:04.020446 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:04.021118 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.021926 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.023156 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.023452 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.023517 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.025190 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:04.025191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:04.025192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:04.025199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:04.026818 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.027695 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.027696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.027696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.027757 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.027965 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.027966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.027966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.028095 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:04.028192 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:04.028433 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:56:04.028520 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:56:04.030828 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } attempt: 3
Sep 25 14:56:04 dist-test-slave-8qt6 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:56:04.039878 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.031123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:36635 (local address 127.0.0.1:33822)
0925 14:56:04.031189 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:04.031200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:04.031237 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:04.031529 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:04.031529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:04.031698 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:04.031771 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:04.031773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.032719 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.032720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:04.033588 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:04.033590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.034343 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.034344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:04.034413 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.034881 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:04.034887 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:04.036595 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:04.038505 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:04.038506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:04.038506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:04.038979 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:04.039197 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:04.039198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:04.039198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:04.039255 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:04.039552 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:04.039553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:04.039601 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:56:04.039675 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:56:04.039775 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:56:04.040185 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:04.031334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.20.199.60:36635)
0925 14:56:04.031393 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:04.031395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:04.031403 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:04.031423 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:04.031423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:04.031439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:04.031513 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:04.031836 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.032632 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.033953 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:04.034263 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:04.034332 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:04.036696 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:04.036697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:04.036698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:04.036705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:04.038408 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.039061 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.039062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.039063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.039120 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:04.039325 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:04.039326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:04.039327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:04.039473 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:04.039545 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:04.039715 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:56:04.039826 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:56:04.042086 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: Bootstrap starting.
I20260925 14:56:04.042831 2262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:04.043124 2262 log.cc:824] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:04.043972 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: No bootstrap required, opened a new log
I20260925 14:56:04.046042 2262 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.046200 2262 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:04.046238 2262 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c3267555f2436388d73505f18d8372, State: Initialized, Role: FOLLOWER
I20260925 14:56:04.046380 2262 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.046705 2262 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:56:04.048518 2391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:56:04.048642 2391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:56:04.046656 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.048832 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:04.050547 2194 master_runner.cc:263] No leader master found from master d6c3267555f2436388d73505f18d8372
I20260925 14:56:04.050594 2194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:56:04.065763 2366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:04.065892 2366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.066125 2366 leader_election.cc:288] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb5069f1368948da93e6706bc83bf08f (127.20.199.62:39759), d6c3267555f2436388d73505f18d8372 (127.20.199.61:37381)
I20260925 14:56:04.066635 2236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3267555f2436388d73505f18d8372" is_pre_election: true
I20260925 14:56:04.066784 2236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 907de4ec7b3b40a7bd94a0b362c754ef in term 0.
I20260925 14:56:04.067062 2161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5069f1368948da93e6706bc83bf08f" is_pre_election: true
I20260925 14:56:04.067215 2161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 907de4ec7b3b40a7bd94a0b362c754ef in term 0.
I20260925 14:56:04.067543 2287 leader_election.cc:302] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 907de4ec7b3b40a7bd94a0b362c754ef, fb5069f1368948da93e6706bc83bf08f; no voters:
I20260925 14:56:04.067771 2366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:04.067821 2366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:04.067844 2366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:04.068771 2366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.068882 2366 leader_election.cc:288] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [CANDIDATE]: Term 1 election: Requested vote from peers fb5069f1368948da93e6706bc83bf08f (127.20.199.62:39759), d6c3267555f2436388d73505f18d8372 (127.20.199.61:37381)
I20260925 14:56:04.069264 2236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3267555f2436388d73505f18d8372"
I20260925 14:56:04.069504 2161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5069f1368948da93e6706bc83bf08f"
I20260925 14:56:04.069574 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:04.070475 2161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 907de4ec7b3b40a7bd94a0b362c754ef in term 1.
I20260925 14:56:04.069342 2236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:04.072054 2236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 907de4ec7b3b40a7bd94a0b362c754ef in term 1.
I20260925 14:56:04.072799 2287 leader_election.cc:302] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 907de4ec7b3b40a7bd94a0b362c754ef, d6c3267555f2436388d73505f18d8372; no voters:
I20260925 14:56:04.073208 2366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:04.073388 2366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [term 1 LEADER]: Becoming Leader. State: Replica: 907de4ec7b3b40a7bd94a0b362c754ef, State: Running, Role: LEADER
I20260925 14:56:04.073496 2366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } }
I20260925 14:56:04.073868 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907de4ec7b3b40a7bd94a0b362c754ef. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.073930 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:04.074265 2395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:04.074501 2395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:04.077231 2236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [term 1 FOLLOWER]: Refusing update from remote peer 907de4ec7b3b40a7bd94a0b362c754ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:04.077561 2161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [term 1 FOLLOWER]: Refusing update from remote peer 907de4ec7b3b40a7bd94a0b362c754ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:04.077973 2366 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:04.079290 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907de4ec7b3b40a7bd94a0b362c754ef. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.079380 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:04.079679 2366 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:04.080888 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907de4ec7b3b40a7bd94a0b362c754ef. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.080974 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:04.083818 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.083902 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:04.084300 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.084376 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:04.084631 2396 mvcc.cc:205] Tried to move back new op lower bound from 7333266080057430016 to 7333266080045641728. Current Snapshot: MvccSnapshot[applied={T|T < 7333266080057430016}]
I20260925 14:56:04.084982 2399 mvcc.cc:205] Tried to move back new op lower bound from 7333266080057430016 to 7333266080045641728. Current Snapshot: MvccSnapshot[applied={T|T < 7333266080057430016}]
I20260925 14:56:04.085388 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.085502 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:04.085903 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb5069f1368948da93e6706bc83bf08f" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 39759 } } peers { permanent_uuid: "d6c3267555f2436388d73505f18d8372" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 37381 } } peers { permanent_uuid: "907de4ec7b3b40a7bd94a0b362c754ef" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 36635 } } }
I20260925 14:56:04.085984 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:56:04.086663 2397 mvcc.cc:205] Tried to move back new op lower bound from 7333266080057430016 to 7333266080045641728. Current Snapshot: MvccSnapshot[applied={T|T < 7333266080057430016}]
I20260925 14:56:04.087291 2395 catalog_manager.cc:1347] Generated new cluster ID: 69f14bb034b745f49483592efb8fbd3d
I20260925 14:56:04.087334 2395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:04.103504 2395 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:04.104337 2395 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:04.112581 2395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907de4ec7b3b40a7bd94a0b362c754ef: Generated new TSK 0
I20260925 14:56:04.112876 2395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:56:05.005151 2356 catalog_manager.cc:1259] Loaded cluster ID: 69f14bb034b745f49483592efb8fbd3d
I20260925 14:56:05.005224 2356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: loading cluster ID for follower catalog manager: success
I20260925 14:56:05.006126 2356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: acquiring CA information for follower catalog manager: success
I20260925 14:56:05.006454 2356 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb5069f1368948da93e6706bc83bf08f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:56:05.049086 2391 catalog_manager.cc:1259] Loaded cluster ID: 69f14bb034b745f49483592efb8fbd3d
I20260925 14:56:05.049160 2391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: loading cluster ID for follower catalog manager: success
I20260925 14:56:05.050006 2391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: acquiring CA information for follower catalog manager: success
I20260925 14:56:05.050312 2391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6c3267555f2436388d73505f18d8372: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:56:05.174468 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2119
I20260925 14:56:05.180064 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2194
W20260925 14:56:05.184834 2287 connection.cc:535] client connection to 127.20.199.61:37381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:56:05.184974 2287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:56:05.185106 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2269
2026-09-25T14:56:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2829 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:05 dist-test-slave-8qt6 krb5kdc[2412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:05 dist-test-slave-8qt6 krb5kdc[2412](info): set up 2 sockets
Sep 25 14:56:05 dist-test-slave-8qt6 krb5kdc[2412](info): commencing operation
krb5kdc: starting...
W20260925 14:56:06.257385 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:56:06 dist-test-slave-8qt6 krb5kdc[2412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:56:06.375826 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:32839
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:32839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:06.456166 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:06.456449 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:06.456489 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:06.457741 2425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:06.457815 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:06.457839 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:06.457870 2425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:06.457899 2425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:06.459410 2425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:32839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:06.459700 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:06.459944 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:06.464957 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:06.464950 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:06.464964 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:06.465286 2425 server_base.cc:1034] running on GCE node
I20260925 14:56:06.465580 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:06.465876 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:06.467052 2425 hybrid_clock.cc:648] HybridClock initialized: now 1790348166467029 us; error 33 us; skew 500 ppm
Sep 25 14:56:06 dist-test-slave-8qt6 krb5kdc[2412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348166, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:06.469569 2425 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:06.470103 2425 webserver.cc:460] Webserver started at http://127.20.199.62:33275/ using document root <none> and password file <none>
I20260925 14:56:06.470337 2425 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:06.470413 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:06.470551 2425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:06.471628 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "0f06c6cea3da4e9c89de0db5c92ab822"
format_stamp: "Formatted at 2026-09-25 14:56:06 on dist-test-slave-8qt6"
server_key: "940f766db195bc12968f8f3ce3506bff"
server_key_iv: "04077b4b22a7fbcc09b89b416b80f422"
server_key_version: "encryptionkey@0"
I20260925 14:56:06.472013 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "0f06c6cea3da4e9c89de0db5c92ab822"
format_stamp: "Formatted at 2026-09-25 14:56:06 on dist-test-slave-8qt6"
server_key: "940f766db195bc12968f8f3ce3506bff"
server_key_iv: "04077b4b22a7fbcc09b89b416b80f422"
server_key_version: "encryptionkey@0"
I20260925 14:56:06.473387 2425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:56:06.474221 2440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:06.474416 2425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:06.474517 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "0f06c6cea3da4e9c89de0db5c92ab822"
format_stamp: "Formatted at 2026-09-25 14:56:06 on dist-test-slave-8qt6"
server_key: "940f766db195bc12968f8f3ce3506bff"
server_key_iv: "04077b4b22a7fbcc09b89b416b80f422"
server_key_version: "encryptionkey@0"
I20260925 14:56:06.474608 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:06.490564 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:06.492586 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:06.492753 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:06.496856 2425 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:32839
I20260925 14:56:06.496912 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:32839 every 8 connection(s)
I20260925 14:56:06.497267 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:06.498175 2493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:06.500384 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822: Bootstrap starting.
I20260925 14:56:06.500489 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2425
I20260925 14:56:06.500581 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:56:06.500761 21276 external_mini_cluster.cc:1428] Setting key be255c479bbf9638bca5a516c97a41d5
I20260925 14:56:06.501080 2493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:06.501688 2493 log.cc:824] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:06.502437 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822: No bootstrap required, opened a new log
I20260925 14:56:06.504792 2493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 32839 } }
I20260925 14:56:06.504954 2493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:06.505000 2493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f06c6cea3da4e9c89de0db5c92ab822, State: Initialized, Role: FOLLOWER
I20260925 14:56:06.505138 2493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 32839 } }
I20260925 14:56:06.505235 2493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:06.505275 2493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:06.505313 2493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:06.505947 2493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 32839 } }
I20260925 14:56:06.506070 2493 leader_election.cc:302] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f06c6cea3da4e9c89de0db5c92ab822; no voters:
I20260925 14:56:06.506263 2493 leader_election.cc:288] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:06.506350 2498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:06.506515 2498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [term 1 LEADER]: Becoming Leader. State: Replica: 0f06c6cea3da4e9c89de0db5c92ab822, State: Running, Role: LEADER
I20260925 14:56:06.506539 2493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:06.506704 2498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 32839 } }
I20260925 14:56:06.507977 2499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 32839 } } }
I20260925 14:56:06.508061 2499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:06.508152 2500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f06c6cea3da4e9c89de0db5c92ab822. Latest consensus state: current_term: 1 leader_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f06c6cea3da4e9c89de0db5c92ab822" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 32839 } } }
I20260925 14:56:06.508467 2500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822 [sys.catalog]: This master's current role is: LEADER
W20260925 14:56:06.508543 2513 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:56:06.508613 2513 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:56:06.508671 2515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 14:56:06 dist-test-slave-8qt6 krb5kdc[2412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:06.508846 2515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:06.510406 2515 catalog_manager.cc:1347] Generated new cluster ID: 65eb4b3c1427427a955cb71c2e331601
I20260925 14:56:06.510458 2515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:06.512794 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:06.501272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.20.199.62:32839)
0925 14:56:06.501461 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:06.501471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:06.501516 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:56:06.502079 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:06.502109 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:06.502172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:06.502321 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:06.502949 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:06.504438 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:06.505491 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:06.506217 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:06.506369 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:06.509247 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:06.509251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:06.509252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:06.509264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:06.511201 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:06.511590 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:06.511593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:06.511594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:06.511691 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:06.511912 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:06.511913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:06.511913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:06.512157 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:06.512332 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:06.512428 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:56:06.512568 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260925 14:56:06.513715 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:56:06.547113 2515 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:06.547762 2515 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:06.552500 2515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f06c6cea3da4e9c89de0db5c92ab822: Generated new TSK 0
I20260925 14:56:06.552702 2515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:56:06.567461 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2425
2026-09-25T14:56:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1380 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:06 dist-test-slave-8qt6 krb5kdc[2520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:06 dist-test-slave-8qt6 krb5kdc[2520](info): set up 2 sockets
Sep 25 14:56:06 dist-test-slave-8qt6 krb5kdc[2520](info): commencing operation
krb5kdc: starting...
W20260925 14:56:07.641645 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:56:07 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:56:07.758710 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34247
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:07.837870 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:07.838119 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:07.838160 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:07.839296 2533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:07.839368 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:07.839393 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:07.839426 2533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:07.839450 2533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:07.841017 2533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:07.841295 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:07.841544 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:07.847007 2538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:07.847007 2539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:07.847067 2541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:07.847373 2533 server_base.cc:1034] running on GCE node
I20260925 14:56:07.847611 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:07.847910 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:07.849083 2533 hybrid_clock.cc:648] HybridClock initialized: now 1790348167849065 us; error 28 us; skew 500 ppm
Sep 25 14:56:07 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:07.851568 2533 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:07.851999 2533 webserver.cc:460] Webserver started at http://127.20.199.62:44621/ using document root <none> and password file <none>
I20260925 14:56:07.852221 2533 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:07.852289 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:07.852440 2533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:07.853639 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "a46d178deeb84b19bfd99e54355de8bf"
format_stamp: "Formatted at 2026-09-25 14:56:07 on dist-test-slave-8qt6"
server_key: "85be351a87865b0c2d9af4d8f2e93b47"
server_key_iv: "a61eca78a948ad8a99b69efae73e7f13"
server_key_version: "encryptionkey@0"
I20260925 14:56:07.854005 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "a46d178deeb84b19bfd99e54355de8bf"
format_stamp: "Formatted at 2026-09-25 14:56:07 on dist-test-slave-8qt6"
server_key: "85be351a87865b0c2d9af4d8f2e93b47"
server_key_iv: "a61eca78a948ad8a99b69efae73e7f13"
server_key_version: "encryptionkey@0"
I20260925 14:56:07.855281 2533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:56:07.855958 2548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:07.856120 2533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:56:07.856248 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "a46d178deeb84b19bfd99e54355de8bf"
format_stamp: "Formatted at 2026-09-25 14:56:07 on dist-test-slave-8qt6"
server_key: "85be351a87865b0c2d9af4d8f2e93b47"
server_key_iv: "a61eca78a948ad8a99b69efae73e7f13"
server_key_version: "encryptionkey@0"
I20260925 14:56:07.856321 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:07.866636 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:07.868434 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:07.868609 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:07.872433 2533 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34247
I20260925 14:56:07.872463 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34247 every 8 connection(s)
I20260925 14:56:07.872828 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:07.873330 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2533
I20260925 14:56:07.873497 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:56:07.873715 21276 external_mini_cluster.cc:1428] Setting key af941f30adac712607b0def2d8c3116d
I20260925 14:56:07.874032 2601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:07.877543 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap starting.
I20260925 14:56:07.878188 2601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:07.878506 2601 log.cc:824] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:07.879283 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: No bootstrap required, opened a new log
Sep 25 14:56:07 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:07.881208 2601 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:07.881438 2601 raft_consensus.cc:374] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:07.881495 2601 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46d178deeb84b19bfd99e54355de8bf, State: Initialized, Role: FOLLOWER
I20260925 14:56:07.881628 2601 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:07.881750 2601 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:07.881783 2601 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:07.881839 2601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:07.882606 2601 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:07.882726 2601 leader_election.cc:302] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46d178deeb84b19bfd99e54355de8bf; no voters:
I20260925 14:56:07.882895 2601 leader_election.cc:288] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:07.882969 2606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:07.883155 2601 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:07.883162 2606 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 LEADER]: Becoming Leader. State: Replica: a46d178deeb84b19bfd99e54355de8bf, State: Running, Role: LEADER
I20260925 14:56:07.883350 2606 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:07.883731 2607 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a46d178deeb84b19bfd99e54355de8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } } }
I20260925 14:56:07.883808 2607 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:07.883965 2608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46d178deeb84b19bfd99e54355de8bf. Latest consensus state: current_term: 1 leader_uuid: "a46d178deeb84b19bfd99e54355de8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } } }
I20260925 14:56:07.884020 2608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:07.884413 2615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:07.884968 2615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:07.885365 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:07.874244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.20.199.62:34247)
0925 14:56:07.874411 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:07.874420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:07.874452 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:56:07.875040 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:07.875050 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:07.875114 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:07.875267 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:07.875847 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.877258 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.878107 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:07.878744 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:07.878829 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:07.881137 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:07.881142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:07.881142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:07.881155 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:07.883292 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.883756 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.883759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.883760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.883894 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:07.884122 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:07.884122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:07.884123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:07.884519 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:07.885012 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:07.885115 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:56:07.885216 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260925 14:56:07.886440 2615 catalog_manager.cc:1347] Generated new cluster ID: 9386ed5c217c4bffa3c39631bc8f953c
I20260925 14:56:07.886512 2615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:07.901357 2615 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:07.902144 2615 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:07.905784 2615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Generated new TSK 0
I20260925 14:56:07.905970 2615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:56:07.937022 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34247
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:08.019299 2627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:08.019513 2627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:08.019595 2627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:08.020797 2627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:08.020841 2627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:08.020866 2627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:56:08.022339 2627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:08.022583 2627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:08.022831 2627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:08.028036 2633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.028081 2632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.028208 2627 server_base.cc:1034] running on GCE node
W20260925 14:56:08.028065 2635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.028476 2627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:08.028697 2627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:08.029829 2627 hybrid_clock.cc:648] HybridClock initialized: now 1790348168029817 us; error 27 us; skew 500 ppm
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:08.032302 2627 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:56:08.032775 2627 webserver.cc:460] Webserver started at http://127.20.199.1:33485/ using document root <none> and password file <none>
I20260925 14:56:08.032997 2627 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:08.033041 2627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:08.033179 2627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:08.034271 2627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "c2e62c939d1c87aed4b58bd0e94fd46f"
server_key_iv: "65e3595bb728bce9a830735a90cf2fb8"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.034639 2627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "c2e62c939d1c87aed4b58bd0e94fd46f"
server_key_iv: "65e3595bb728bce9a830735a90cf2fb8"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.035933 2627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:56:08.036624 2642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.036800 2627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:56:08.036896 2627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "c2e62c939d1c87aed4b58bd0e94fd46f"
server_key_iv: "65e3595bb728bce9a830735a90cf2fb8"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.036978 2627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:08.066395 2627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:08.068249 2627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:08.068431 2627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:08.068686 2627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:08.069007 2627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:08.069061 2627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.069098 2627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:08.069144 2627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.075033 2627 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:43237
I20260925 14:56:08.075075 2755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:43237 every 8 connection(s)
I20260925 14:56:08.075397 2627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:08.083316 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2627
I20260925 14:56:08.083436 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:08.083675 21276 external_mini_cluster.cc:1428] Setting key e8cc06b9b736ad84fe9fa1fac365fe45
I20260925 14:56:08.084712 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.076134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:52155 (local address 127.20.199.62:34247)
0925 14:56:08.076215 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.076217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.076388 (+ 171us) server_negotiation.cc:407] Connection header received
0925 14:56:08.076929 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.076931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.076941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.076991 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.077990 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.078492 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.079466 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.079655 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.079704 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.081862 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.081863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.081863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.081870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.083188 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.083664 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.083665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.083666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.083748 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.084021 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.084022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.084022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.084144 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.084223 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.084532 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.084620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:56:08.085053 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.076085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:52155)
0925 14:56:08.076263 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.076302 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.076842 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.077052 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.077055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.077709 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:08.077910 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.077915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.078593 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.078594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.079387 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.079389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.079724 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.079725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.079825 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.080378 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:08.080387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:08.081771 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:08.083259 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.083268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.083270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.083582 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.083820 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.083821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.083821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.083946 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.084252 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:08.084258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:08.084393 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.084499 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.084657 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260925 14:56:08.085520 2756 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34247
I20260925 14:56:08.085620 2756 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:08.085844 2756 heartbeater.cc:511] Master 127.20.199.62:34247 requested a full tablet report, sending...
I20260925 14:56:08.086365 2565 ts_manager.cc:194] Registered new tserver with Master: 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237)
I20260925 14:56:08.086971 2565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:52155
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:56:08.108767 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34247
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:08.188752 2761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:08.188958 2761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:08.188982 2761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:08.190199 2761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:08.190244 2761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:08.190270 2761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:56:08.191689 2761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:08.191872 2761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:08.192113 2761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:08.197046 2769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.197054 2767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.197046 2766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.197418 2761 server_base.cc:1034] running on GCE node
I20260925 14:56:08.197609 2761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:08.197818 2761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:08.198961 2761 hybrid_clock.cc:648] HybridClock initialized: now 1790348168198953 us; error 30 us; skew 500 ppm
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:08.201326 2761 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:56:08.201800 2761 webserver.cc:460] Webserver started at http://127.20.199.2:33817/ using document root <none> and password file <none>
I20260925 14:56:08.202045 2761 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:08.202116 2761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:08.202284 2761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:08.203330 2761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "a36aeb434d33427a8e59c678f9e111bf"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "8e60c47004b8507269417e5d805edc15"
server_key_iv: "22cdf0323b24ebc6b9de220cc6fae728"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.203704 2761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "a36aeb434d33427a8e59c678f9e111bf"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "8e60c47004b8507269417e5d805edc15"
server_key_iv: "22cdf0323b24ebc6b9de220cc6fae728"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.205016 2761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:56:08.205804 2776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.205986 2761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.206076 2761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "a36aeb434d33427a8e59c678f9e111bf"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "8e60c47004b8507269417e5d805edc15"
server_key_iv: "22cdf0323b24ebc6b9de220cc6fae728"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.206157 2761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:08.224220 2761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:08.226152 2761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:08.226320 2761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:08.226569 2761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:08.226881 2761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:08.226935 2761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.226975 2761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:08.227005 2761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.232959 2761 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44161
I20260925 14:56:08.233001 2889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44161 every 8 connection(s)
I20260925 14:56:08.233342 2761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:08.242724 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.234113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:60349 (local address 127.20.199.62:34247)
0925 14:56:08.234209 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.234211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.234405 (+ 194us) server_negotiation.cc:407] Connection header received
0925 14:56:08.234936 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.234937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.234949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.235003 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.235955 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.236465 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.237528 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.237745 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.237831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.239919 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.239920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.239921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.239928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.241299 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.241748 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.241749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.241750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.241812 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.242082 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.242083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.242084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.242200 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.242275 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.242547 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.242630 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:56:08.242910 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.234082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:60349)
0925 14:56:08.234258 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.234306 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.234828 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.235084 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.235088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.235656 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:08.235848 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.235853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.236562 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.236563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.237367 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.237369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.237842 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.237843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.237973 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.238419 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:08.238427 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:08.239822 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:08.241438 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.241446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.241449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.241670 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.241896 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.241897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.241897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.242008 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.242266 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:08.242273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:08.242415 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.242526 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.242683 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 14:56:08.243551 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2761
I20260925 14:56:08.243575 2890 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34247
I20260925 14:56:08.243649 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:08.243666 2890 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:08.243845 21276 external_mini_cluster.cc:1428] Setting key a44aee5a2e927a58436b5477aa74f63f
I20260925 14:56:08.243883 2890 heartbeater.cc:511] Master 127.20.199.62:34247 requested a full tablet report, sending...
I20260925 14:56:08.244284 2564 ts_manager.cc:194] Registered new tserver with Master: a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
I20260925 14:56:08.244685 2564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:60349
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:56:08.268676 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34247
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:08.348066 2895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:08.348382 2895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:08.348423 2895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:08.349610 2895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:08.349676 2895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:08.349725 2895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:56:08.351145 2895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:08.351404 2895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:08.351644 2895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:08.356616 2900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.356715 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.356739 2895 server_base.cc:1034] running on GCE node
W20260925 14:56:08.356614 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.357036 2895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:08.357277 2895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:08.358441 2895 hybrid_clock.cc:648] HybridClock initialized: now 1790348168358428 us; error 23 us; skew 500 ppm
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:08.360783 2895 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:56:08.361222 2895 webserver.cc:460] Webserver started at http://127.20.199.3:34739/ using document root <none> and password file <none>
I20260925 14:56:08.361446 2895 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:08.361501 2895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:08.361589 2895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:08.362552 2895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "24900d3d26f949058067498ad7d86da1"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "15fb3d71cf499ee098104896299d046d"
server_key_iv: "b4c19aad436f6af0de183335e0588bbb"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.362859 2895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "24900d3d26f949058067498ad7d86da1"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "15fb3d71cf499ee098104896299d046d"
server_key_iv: "b4c19aad436f6af0de183335e0588bbb"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.364113 2895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:56:08.364787 2910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.364984 2895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:56:08.365072 2895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "24900d3d26f949058067498ad7d86da1"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "15fb3d71cf499ee098104896299d046d"
server_key_iv: "b4c19aad436f6af0de183335e0588bbb"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.365142 2895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:08.374069 2895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:08.375742 2895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:08.375896 2895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:08.376106 2895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:08.376390 2895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:08.376423 2895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.376443 2895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:08.376489 2895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.382513 2895 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:34053
I20260925 14:56:08.382555 3023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:34053 every 8 connection(s)
I20260925 14:56:08.382930 2895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:08.383282 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 2895
I20260925 14:56:08.383395 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:08.383597 21276 external_mini_cluster.cc:1428] Setting key 3fd1175be563b4cab23a62bc03b72e47
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:08.392460 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.383635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:34783 (local address 127.20.199.62:34247)
0925 14:56:08.383774 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.383776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.383932 (+ 156us) server_negotiation.cc:407] Connection header received
0925 14:56:08.384652 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.384653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.384663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.384701 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.385726 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.386209 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.387359 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.387549 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.387619 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.389736 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.389737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.389738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.389745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.391278 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.391627 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.391628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.391628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.391700 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.391929 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.391930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.391930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.392049 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.392126 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.392335 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.392398 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 14:56:08.392664 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.383632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:34783)
0925 14:56:08.383783 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.383827 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.384555 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.384775 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.384779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.385447 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:08.385649 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.385654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.386284 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.386284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.387261 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.387263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.387630 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.387631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.387727 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.388199 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:08.388207 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:08.389625 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:08.391362 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.391366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.391368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.391561 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.391759 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.391759 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.391760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.391851 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.392126 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:08.392130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:08.392230 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.392306 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.392424 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260925 14:56:08.393177 3024 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34247
I20260925 14:56:08.393255 3024 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:08.393443 3024 heartbeater.cc:511] Master 127.20.199.62:34247 requested a full tablet report, sending...
I20260925 14:56:08.393906 2564 ts_manager.cc:194] Registered new tserver with Master: 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053)
I20260925 14:56:08.394292 2564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:34783
I20260925 14:56:08.395282 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:08.402501 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.396594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.20.199.62:34247)
0925 14:56:08.396670 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.396672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.396779 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:56:08.396863 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.396864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.396874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.396929 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.397494 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.397952 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.398722 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.398915 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.398989 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.400180 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.400181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.400181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.400188 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.401319 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.401691 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.401692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.401693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.401751 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.401944 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.401945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.401945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.402107 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.402219 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.402327 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.402405 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:56:08.404251 2564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:08.405946 2564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:08.412508 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.408207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44161 (local address 127.0.0.1:35060)
0925 14:56:08.408592 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.408603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.408680 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.408950 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.408951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.408952 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.408992 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.408994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.410071 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.410071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.411449 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.411452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.412154 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.412155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.412250 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.412270 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.412368 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.412426 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":46,"thread_start_us":82,"threads_started":1}
I20260925 14:56:08.412508 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.408523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.20.199.2:44161)
0925 14:56:08.408722 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.408728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.408738 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:56:08.408779 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.408781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.408796 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.408961 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.409100 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.409958 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.411554 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.412049 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.412151 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.412213 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.412387 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.412428 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260925 14:56:08.412704 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.408207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43237 (local address 127.0.0.1:45468)
0925 14:56:08.408388 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.408428 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.408502 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.408777 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.408779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.408781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.408868 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.408870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.409969 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.409970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.411449 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.411452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.412445 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.412446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.412507 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.412511 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.412557 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.412610 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":55,"thread_start_us":51,"threads_started":1}
I20260925 14:56:08.412930 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.408315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.20.199.1:43237)
0925 14:56:08.408490 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.408495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.408508 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:56:08.408641 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.408644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.408699 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.408776 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.408937 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.409870 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.411536 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.412294 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.412362 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.412411 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.412810 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.412854 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260925 14:56:08.413375 2824 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f (DEFAULT_TABLE table=test-table [id=46abde41f5dc4125b46ce0e51b5d7986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:08.413601 2690 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f (DEFAULT_TABLE table=test-table [id=46abde41f5dc4125b46ce0e51b5d7986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:08.413810 2824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:08.413933 2690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:08.414332 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.408399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34053 (local address 127.0.0.1:55800)
0925 14:56:08.410086 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.410094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.410173 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.410354 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.410355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.410356 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.410407 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.410409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.411390 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.411391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.412493 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.412494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.414089 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.414090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.414150 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.414154 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.414213 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.414257 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"thread_start_us":68,"threads_started":1}
I20260925 14:56:08.414523 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.408585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.20.199.3:34053)
0925 14:56:08.408758 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.408761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.410152 (+ 1391us) server_negotiation.cc:407] Connection header received
0925 14:56:08.410251 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.410253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.410268 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.410357 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.410462 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.411293 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.412942 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.413928 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.413993 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.414049 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.414244 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.414294 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260925 14:56:08.415287 2958 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f (DEFAULT_TABLE table=test-table [id=46abde41f5dc4125b46ce0e51b5d7986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:08.415583 2958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:08.416713 3041 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap starting.
I20260925 14:56:08.417080 3042 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Bootstrap starting.
I20260925 14:56:08.417380 3041 tablet_bootstrap.cc:654] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:08.417761 3042 tablet_bootstrap.cc:654] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:08.417979 3041 log.cc:824] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:08.418372 3042 log.cc:824] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:08.418830 3045 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Bootstrap starting.
I20260925 14:56:08.419059 3041 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: No bootstrap required, opened a new log
I20260925 14:56:08.419137 3041 ts_tablet_manager.cc:1397] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:56:08.419166 3042 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: No bootstrap required, opened a new log
I20260925 14:56:08.419234 3042 ts_tablet_manager.cc:1397] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:08.419479 3045 tablet_bootstrap.cc:654] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:08.420008 3045 log.cc:824] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:08.420797 3045 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: No bootstrap required, opened a new log
I20260925 14:56:08.420871 3045 ts_tablet_manager.cc:1397] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:08.420886 3042 raft_consensus.cc:348] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.421056 3042 raft_consensus.cc:374] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:08.421097 3042 raft_consensus.cc:729] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36aeb434d33427a8e59c678f9e111bf, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.421218 3042 consensus_queue.cc:260] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.421231 3041 raft_consensus.cc:348] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.421420 3041 raft_consensus.cc:374] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:08.421473 3041 raft_consensus.cc:729] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b05c69a8e8a499d896ae3ab6ec951fb, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.421530 3042 ts_tablet_manager.cc:1428] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:56:08.421583 3041 consensus_queue.cc:260] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.421689 2890 heartbeater.cc:503] Master 127.20.199.62:34247 was elected leader, sending a full tablet report...
I20260925 14:56:08.421856 3041 ts_tablet_manager.cc:1428] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:08.421867 2756 heartbeater.cc:503] Master 127.20.199.62:34247 was elected leader, sending a full tablet report...
I20260925 14:56:08.422915 3045 raft_consensus.cc:348] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.423041 3045 raft_consensus.cc:374] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:08.423063 3045 raft_consensus.cc:729] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24900d3d26f949058067498ad7d86da1, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.423226 3045 consensus_queue.cc:260] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.423504 3045 ts_tablet_manager.cc:1428] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:08.423521 3024 heartbeater.cc:503] Master 127.20.199.62:34247 was elected leader, sending a full tablet report...
W20260925 14:56:08.483939 2891 tablet.cc:2369] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:08.492035 3047 raft_consensus.cc:482] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:08.492161 3047 raft_consensus.cc:504] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.492539 3047 leader_election.cc:288] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237), 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053)
I20260925 14:56:08.496636 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.492849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:45537 (local address 127.20.199.3:34053)
0925 14:56:08.492930 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.492932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.493088 (+ 156us) server_negotiation.cc:407] Connection header received
0925 14:56:08.493149 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.493150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.493159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.493211 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.493350 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.494120 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.495647 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.496216 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.496291 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.496300 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.496517 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.496558 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:56:08.496636 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.492922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:55757 (local address 127.20.199.1:43237)
0925 14:56:08.493009 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.493012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.493023 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:56:08.493073 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.493074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.493088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.493144 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.493352 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.494116 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.495620 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.496216 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.496292 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.496300 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.496511 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.496565 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:56:08.496663 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.492742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43237 (local address 127.20.199.2:55757)
0925 14:56:08.492809 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.492823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.492936 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.493162 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.493163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.493165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.493256 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.493258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.494208 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.494208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.495520 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.495523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.496310 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.496311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.496394 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.496396 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.496472 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.496528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":48}
I20260925 14:56:08.496675 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.492742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34053 (local address 127.20.199.2:45537)
0925 14:56:08.493030 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.493041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.493095 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.493221 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.493222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.493223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.493276 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.493277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.494208 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.494208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.495571 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.495572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.496310 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.496311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.496390 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.496394 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.496472 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.496528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":122,"thread_start_us":115,"threads_started":1}
I20260925 14:56:08.497169 2978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ef9d5c5ed47e29484e7770a80ae6f" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24900d3d26f949058067498ad7d86da1" is_pre_election: true
I20260925 14:56:08.497169 2710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ef9d5c5ed47e29484e7770a80ae6f" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" is_pre_election: true
I20260925 14:56:08.497334 2710 raft_consensus.cc:2454] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 0.
I20260925 14:56:08.497334 2978 raft_consensus.cc:2454] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 0.
I20260925 14:56:08.497630 2779 leader_election.cc:302] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb, a36aeb434d33427a8e59c678f9e111bf; no voters:
I20260925 14:56:08.497795 3047 raft_consensus.cc:2781] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:08.497856 3047 raft_consensus.cc:482] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:08.497879 3047 raft_consensus.cc:3037] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:08.498615 3047 raft_consensus.cc:504] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.498769 3047 leader_election.cc:288] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 election: Requested vote from peers 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237), 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053)
I20260925 14:56:08.498953 2710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ef9d5c5ed47e29484e7770a80ae6f" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
I20260925 14:56:08.498953 2978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ef9d5c5ed47e29484e7770a80ae6f" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24900d3d26f949058067498ad7d86da1"
I20260925 14:56:08.499034 2978 raft_consensus.cc:3037] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:08.499035 2710 raft_consensus.cc:3037] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:08.499938 2710 raft_consensus.cc:2454] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 1.
I20260925 14:56:08.499938 2978 raft_consensus.cc:2454] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 1.
I20260925 14:56:08.500116 2779 leader_election.cc:302] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb, a36aeb434d33427a8e59c678f9e111bf; no voters:
I20260925 14:56:08.500213 3047 raft_consensus.cc:2781] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:08.500375 3047 raft_consensus.cc:686] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 1 LEADER]: Becoming Leader. State: Replica: a36aeb434d33427a8e59c678f9e111bf, State: Running, Role: LEADER
I20260925 14:56:08.500470 3047 consensus_queue.cc:237] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.501456 2565 catalog_manager.cc:5495] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf reported cstate change: term changed from 0 to 1, leader changed from <none> to a36aeb434d33427a8e59c678f9e111bf (127.20.199.2). New cstate: current_term: 1 leader_uuid: "a36aeb434d33427a8e59c678f9e111bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:08.528226 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.526126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.20.199.2:44161)
0925 14:56:08.526191 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.526193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.526202 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:56:08.526249 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.526250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.526268 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.526327 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:08.526444 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.526890 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.527571 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.527752 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.527804 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.527953 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:08.528043 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:08.528103 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.528136 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.528152 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:56:08.529997 2710 raft_consensus.cc:1261] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Refusing update from remote peer a36aeb434d33427a8e59c678f9e111bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:08.530022 2978 raft_consensus.cc:1261] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Refusing update from remote peer a36aeb434d33427a8e59c678f9e111bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:08.530365 3047 consensus_queue.cc:1035] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:08.530516 3051 consensus_queue.cc:1035] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:08.532610 3054 mvcc.cc:205] Tried to move back new op lower bound from 7333266098296016896 to 7333266098178547712. Current Snapshot: MvccSnapshot[applied={T|T < 7333266098296016896}]
I20260925 14:56:08.533890 3055 mvcc.cc:205] Tried to move back new op lower bound from 7333266098296016896 to 7333266098178547712. Current Snapshot: MvccSnapshot[applied={T|T < 7333266098296016896 or (T in {7333266098296016896})}]
I20260925 14:56:08.534017 3056 mvcc.cc:205] Tried to move back new op lower bound from 7333266098296016896 to 7333266098178547712. Current Snapshot: MvccSnapshot[applied={T|T < 7333266098296016896 or (T in {7333266098296016896})}]
I20260925 14:56:08.535118 2564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:08.535211 2564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:08.535892 2564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Sending DeleteTablet for 3 replicas of tablet 1b8ef9d5c5ed47e29484e7770a80ae6f
I20260925 14:56:08.536226 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2627
I20260925 14:56:08.536221 2690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:45468
I20260925 14:56:08.536218 2958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:55800
I20260925 14:56:08.536222 2824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:35060
I20260925 14:56:08.537139 3065 tablet_replica.cc:330] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: stopping tablet replica
I20260925 14:56:08.537248 3065 raft_consensus.cc:2229] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:08.537353 3065 raft_consensus.cc:2258] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:08.537534 3064 tablet_replica.cc:330] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: stopping tablet replica
I20260925 14:56:08.537624 3064 raft_consensus.cc:2229] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:08.537716 3064 raft_consensus.cc:2258] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:08.538130 3065 ts_tablet_manager.cc:1905] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:08.539906 3065 ts_tablet_manager.cc:1918] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:08.540033 3065 log.cc:1196] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/1b8ef9d5c5ed47e29484e7770a80ae6f
I20260925 14:56:08.540084 3064 ts_tablet_manager.cc:1905] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:08.540424 3065 ts_tablet_manager.cc:1939] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 24900d3d26f949058067498ad7d86da1: Deleting consensus metadata
I20260925 14:56:08.540890 2552 catalog_manager.cc:4841] TS 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): tablet 1b8ef9d5c5ed47e29484e7770a80ae6f (table test-table [id=46abde41f5dc4125b46ce0e51b5d7986]) successfully deleted
I20260925 14:56:08.541678 3064 ts_tablet_manager.cc:1918] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:08.541749 3064 log.cc:1196] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/1b8ef9d5c5ed47e29484e7770a80ae6f
I20260925 14:56:08.542026 3064 ts_tablet_manager.cc:1939] T 1b8ef9d5c5ed47e29484e7770a80ae6f P a36aeb434d33427a8e59c678f9e111bf: Deleting consensus metadata
W20260925 14:56:08.542411 2551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:43237 (error 108)
I20260925 14:56:08.542620 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2761
I20260925 14:56:08.543197 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.542798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43237 (local address 127.0.0.1:45476)
0925 14:56:08.543122 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.543163 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302}
W20260925 14:56:08.543349 2551 catalog_manager.cc:4584] TS 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43237: connect: Connection refused (error 111)
I20260925 14:56:08.547890 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2895
I20260925 14:56:08.547969 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.547860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44161 (local address 127.0.0.1:35076)
0925 14:56:08.547916 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.547944 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.2:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:56:08.552206 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 2533
I20260925 14:56:08.557277 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34247
--webserver_interface=127.20.199.62
--webserver_port=44621
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:08.638005 3067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:08.638206 3067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:08.638227 3067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:08.639379 3067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:08.639421 3067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:08.639436 3067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:08.639448 3067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:08.639503 3067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:08.640933 3067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:08.641175 3067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:08.641444 3067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:08.649844 3076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.649844 3074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.649861 3073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.650516 3067 server_base.cc:1034] running on GCE node
I20260925 14:56:08.650807 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:08.651126 3067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:08.652288 3067 hybrid_clock.cc:648] HybridClock initialized: now 1790348168652267 us; error 33 us; skew 500 ppm
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:08.660969 3067 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:08.661389 3067 webserver.cc:460] Webserver started at http://127.20.199.62:44621/ using document root <none> and password file <none>
I20260925 14:56:08.661715 3067 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:08.661772 3067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:08.663095 3067 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:08.663695 3083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.663888 3067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:56:08.663965 3067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "a46d178deeb84b19bfd99e54355de8bf"
format_stamp: "Formatted at 2026-09-25 14:56:07 on dist-test-slave-8qt6"
server_key: "85be351a87865b0c2d9af4d8f2e93b47"
server_key_iv: "a61eca78a948ad8a99b69efae73e7f13"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.664331 3067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:08.688058 3067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:08.690160 3067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:08.690279 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:08.694227 3067 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34247
I20260925 14:56:08.694257 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34247 every 8 connection(s)
I20260925 14:56:08.694690 3067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:08.696472 3136 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:56:08.697081 3136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap starting.
I20260925 14:56:08.702273 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3067
I20260925 14:56:08.702555 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:43237
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:34247
--webserver_port=33485
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:08.705129 3136 log.cc:824] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:08.707047 3136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:08.707347 3136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap complete.
I20260925 14:56:08.709717 3136 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:08.709975 3136 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46d178deeb84b19bfd99e54355de8bf, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.710176 3136 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:08.710271 3136 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:08.710330 3136 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:08.710377 3136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:56:08.711350 3136 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:08.711470 3136 leader_election.cc:302] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46d178deeb84b19bfd99e54355de8bf; no voters:
I20260925 14:56:08.711673 3136 leader_election.cc:288] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:56:08.711707 3140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:56:08.711815 3140 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 LEADER]: Becoming Leader. State: Replica: a46d178deeb84b19bfd99e54355de8bf, State: Running, Role: LEADER
I20260925 14:56:08.711900 3140 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:08.711947 3136 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:08.712172 3140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46d178deeb84b19bfd99e54355de8bf. Latest consensus state: current_term: 2 leader_uuid: "a46d178deeb84b19bfd99e54355de8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } } }
I20260925 14:56:08.712244 3140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:08.712175 3141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a46d178deeb84b19bfd99e54355de8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } } }
I20260925 14:56:08.712281 3141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:08.712831 3144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:08.713599 3144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:08.713687 3144 catalog_manager.cc:1259] Loaded cluster ID: 9386ed5c217c4bffa3c39631bc8f953c
I20260925 14:56:08.713974 3144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:08.714260 3144 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 14:56:08.718961 3144 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 14:56:08.721282 3144 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:08.721479 3144 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Loaded TSK: 0
I20260925 14:56:08.721688 3144 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:56:08.785802 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:08.786007 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:08.786029 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:08.787241 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:08.787281 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:08.787307 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:56:08.788726 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:43237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=33485
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:08.788964 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:08.789216 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:08.795177 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.795413 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.795183 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.795393 3138 server_base.cc:1034] running on GCE node
I20260925 14:56:08.795722 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:08.795992 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:08.797169 3138 hybrid_clock.cc:648] HybridClock initialized: now 1790348168797147 us; error 40 us; skew 500 ppm
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:08.800130 3138 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:56:08.800644 3138 webserver.cc:460] Webserver started at http://127.20.199.1:33485/ using document root <none> and password file <none>
I20260925 14:56:08.800904 3138 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:08.800988 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:08.802698 3138 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:08.803429 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.803607 3138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:56:08.803682 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "c2e62c939d1c87aed4b58bd0e94fd46f"
server_key_iv: "65e3595bb728bce9a830735a90cf2fb8"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.803994 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:08.812583 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:08.814745 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:08.814880 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:08.815114 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:08.815572 3180 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:56:08.817389 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:56:08.817514 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:56:08.817555 3138 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:56:08.818150 3138 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:56:08.818204 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:08.818269 3180 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap starting.
I20260925 14:56:08.825484 3138 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:43237
I20260925 14:56:08.825875 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:08.827492 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3138
I20260925 14:56:08.827773 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:44161
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:34247
--webserver_port=33817
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:08.829015 3180 log.cc:824] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:08.830094 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:43237 every 8 connection(s)
I20260925 14:56:08.831738 3180 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:08.832053 3180 tablet_bootstrap.cc:492] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap complete.
I20260925 14:56:08.833267 3180 ts_tablet_manager.cc:1397] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260925 14:56:08.834623 3180 raft_consensus.cc:348] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.834940 3180 raft_consensus.cc:729] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b05c69a8e8a499d896ae3ab6ec951fb, State: Initialized, Role: FOLLOWER
I20260925 14:56:08.835095 3180 consensus_queue.cc:260] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } } peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } }
I20260925 14:56:08.835378 3180 ts_tablet_manager.cc:1428] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:08.844383 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.826585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:41857)
0925 14:56:08.829446 (+ 2861us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.829496 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.832548 (+ 3052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.833145 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.833149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.833570 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:08.833864 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.833871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.835617 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.835618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.836809 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.836812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.837780 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.837781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.837918 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.838557 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:08.838567 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:08.840418 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:08.842625 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.842631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.842633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.842932 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.843212 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.843213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.843213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.843343 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.843767 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:08.843772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:08.843915 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.844023 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.844177 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2766,"thread_start_us":99,"threads_started":1}
I20260925 14:56:08.844735 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.826661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:41857 (local address 127.20.199.62:34247)
0925 14:56:08.830377 (+ 3716us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.830386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.830417 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:56:08.832824 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.832831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.832891 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.833011 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.834011 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.835522 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.836926 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.837666 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.837780 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.840534 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.840538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.840538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.840549 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.842543 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.843015 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.843019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.843019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.843138 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.843422 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.843422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.843423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.843692 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.843776 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.844090 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.844219 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3662,"thread_start_us":79,"threads_started":1}
I20260925 14:56:08.845373 3288 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34247
I20260925 14:56:08.845506 3288 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:08.845688 3288 heartbeater.cc:511] Master 127.20.199.62:34247 requested a full tablet report, sending...
I20260925 14:56:08.846643 3100 ts_manager.cc:194] Registered new tserver with Master: 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237)
I20260925 14:56:08.847800 3100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:41857
I20260925 14:56:08.848726 3288 heartbeater.cc:503] Master 127.20.199.62:34247 was elected leader, sending a full tablet report...
I20260925 14:56:08.853643 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.848083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.20.199.1:43237)
0925 14:56:08.849508 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.849513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.849610 (+ 97us) server_negotiation.cc:407] Connection header received
0925 14:56:08.849701 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.849704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.849719 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.849796 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:08.849960 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.851004 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.852423 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.853122 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.853193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.853238 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.853533 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.853573 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":97,"threads_started":1}
I20260925 14:56:08.853749 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.847260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43237 (local address 127.0.0.1:45478)
0925 14:56:08.849504 (+ 2244us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.849542 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.849611 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.849817 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.849820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.849823 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:08.849882 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.849884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.851086 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.851087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.852329 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.852332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.853312 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.853313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.853394 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.853408 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.853516 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.853647 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2215,"thread_start_us":85,"threads_started":1}
I20260925 14:56:08.854336 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ef9d5c5ed47e29484e7770a80ae6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:08 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:45478
I20260925 14:56:08.854983 3297 tablet_replica.cc:330] stopping tablet replica
I20260925 14:56:08.855086 3297 raft_consensus.cc:2229] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:08.855163 3297 raft_consensus.cc:2258] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:08.855464 3297 ts_tablet_manager.cc:1905] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:08.857721 3297 ts_tablet_manager.cc:1918] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:08.857796 3297 log.cc:1196] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/1b8ef9d5c5ed47e29484e7770a80ae6f
I20260925 14:56:08.858026 3297 ts_tablet_manager.cc:1939] T 1b8ef9d5c5ed47e29484e7770a80ae6f P 5b05c69a8e8a499d896ae3ab6ec951fb: Deleting consensus metadata
I20260925 14:56:08.858537 3086 catalog_manager.cc:4841] TS 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237): tablet 1b8ef9d5c5ed47e29484e7770a80ae6f (table test-table [id=46abde41f5dc4125b46ce0e51b5d7986]) successfully deleted
W20260925 14:56:08.919142 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:08.919356 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:08.919389 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:08.920496 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:08.920543 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:08.920569 3292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:56:08.921945 3292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:44161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=33817
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:08.922122 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:08.922292 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:08.927239 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:08.927258 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.927433 3292 server_base.cc:1034] running on GCE node
W20260925 14:56:08.927246 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:08.927729 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:08.927996 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:08.929179 3292 hybrid_clock.cc:648] HybridClock initialized: now 1790348168929166 us; error 28 us; skew 500 ppm
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:08.931727 3292 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:56:08.932178 3292 webserver.cc:460] Webserver started at http://127.20.199.2:33817/ using document root <none> and password file <none>
I20260925 14:56:08.932428 3292 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:08.932493 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:08.933772 3292 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:08.934333 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.934531 3292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:56:08.934643 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "a36aeb434d33427a8e59c678f9e111bf"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "8e60c47004b8507269417e5d805edc15"
server_key_iv: "22cdf0323b24ebc6b9de220cc6fae728"
server_key_version: "encryptionkey@0"
I20260925 14:56:08.935027 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:08.952409 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:08.954487 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:08.954602 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:08.954799 3292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:08.955138 3292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:08.955180 3292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.955204 3292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:08.955252 3292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:08.961264 3292 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44161
I20260925 14:56:08.961345 3425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44161 every 8 connection(s)
I20260925 14:56:08.961740 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:08.965276 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3292
I20260925 14:56:08.965595 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:34053
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:34247
--webserver_port=34739
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:56:08 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:08.971954 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.962539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:60809 (local address 127.20.199.62:34247)
0925 14:56:08.962601 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:08.962603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:08.962854 (+ 251us) server_negotiation.cc:407] Connection header received
0925 14:56:08.963352 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:08.963353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:08.963362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:08.963422 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:08.964371 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.964856 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.966388 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.966582 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.966664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.968953 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:08.968955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:08.968955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:08.968963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:08.970483 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.970950 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.970952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.970952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.971014 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:08.971270 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:08.971271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:08.971271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:08.971394 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:08.971467 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:08.971767 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 14:56:08.971861 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:56:08.972142 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:08.962500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:60809)
0925 14:56:08.962678 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:08.962723 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:08.963262 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:08.963499 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:08.963502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:08.964090 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:08.964293 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.964298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.964938 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.964939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.966305 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:08.966309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:08.966667 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:08.966669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:08.966816 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:08.967396 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:08.967406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:08.968846 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:08.970586 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.970593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.970595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.970851 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.971082 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:08.971083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:08.971084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:08.971198 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:08.971472 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:08.971479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:08.971619 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:56:08.971743 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:56:08.971900 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260925 14:56:08.972642 3426 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34247
I20260925 14:56:08.972738 3426 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:08.972941 3426 heartbeater.cc:511] Master 127.20.199.62:34247 requested a full tablet report, sending...
I20260925 14:56:08.973363 3100 ts_manager.cc:194] Registered new tserver with Master: a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
I20260925 14:56:08.973973 3100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:60809
W20260925 14:56:09.062426 3429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:09.062635 3429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:09.062659 3429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:09.063766 3429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:09.063802 3429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:09.063827 3429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:56:09.065219 3429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=34739
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:09.065474 3429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:09.065672 3429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:09.070640 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.070633 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.070633 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.070811 3429 server_base.cc:1034] running on GCE node
I20260925 14:56:09.071059 3429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:09.071306 3429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:09.072450 3429 hybrid_clock.cc:648] HybridClock initialized: now 1790348169072439 us; error 26 us; skew 500 ppm
Sep 25 14:56:09 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:09.074888 3429 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:56:09.075307 3429 webserver.cc:460] Webserver started at http://127.20.199.3:34739/ using document root <none> and password file <none>
I20260925 14:56:09.075544 3429 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:09.075624 3429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:09.076908 3429 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:09.077513 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.077677 3429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.077773 3429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "24900d3d26f949058067498ad7d86da1"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "15fb3d71cf499ee098104896299d046d"
server_key_iv: "b4c19aad436f6af0de183335e0588bbb"
server_key_version: "encryptionkey@0"
I20260925 14:56:09.078114 3429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:09.092195 3429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:09.094095 3429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:09.094259 3429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:09.094487 3429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:09.094794 3429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:09.094852 3429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.094893 3429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:09.094919 3429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.100834 3429 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:34053
I20260925 14:56:09.100893 3557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:34053 every 8 connection(s)
I20260925 14:56:09.101225 3429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:56:09 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:09.110741 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.102226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:60119 (local address 127.20.199.62:34247)
0925 14:56:09.102294 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.102297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.102479 (+ 182us) server_negotiation.cc:407] Connection header received
0925 14:56:09.103005 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.103007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.103020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.103069 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:09.104016 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.104515 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.105563 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.105774 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.105887 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.107969 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:09.107970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:09.107971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:09.107978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:09.109209 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:09.109690 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:09.109692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:09.109692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:09.109753 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:09.110003 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:09.110004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:09.110005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:09.110121 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:09.110204 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.110526 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.110627 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:56:09.110994 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.102198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:60119)
0925 14:56:09.102341 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.102395 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:09.102902 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:09.103152 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:09.103158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:09.103724 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:09.103916 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.103921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.104611 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.104612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.105478 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.105480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.105896 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.105898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.106038 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.106557 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:09.106564 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:09.107872 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:09.109291 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:09.109297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:09.109300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:09.109602 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:09.109819 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:09.109820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:09.109820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:09.109934 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:09.110208 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:09.110215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:09.110373 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 14:56:09.110509 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:56:09.110681 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260925 14:56:09.111290 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3429
I20260925 14:56:09.111529 3558 heartbeater.cc:347] Connected to a master server at 127.20.199.62:34247
I20260925 14:56:09.111649 3558 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:09.111912 3558 heartbeater.cc:511] Master 127.20.199.62:34247 requested a full tablet report, sending...
I20260925 14:56:09.112355 3100 ts_manager.cc:194] Registered new tserver with Master: 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053)
I20260925 14:56:09.112893 3100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:60119
I20260925 14:56:09.117782 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.111851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.20.199.62:34247)
0925 14:56:09.111889 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.111891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.111990 (+ 99us) server_negotiation.cc:407] Connection header received
0925 14:56:09.112060 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.112061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.112071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.112121 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:09.112973 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.113691 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.114423 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.114590 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.114640 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.115752 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:09.115753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:09.115753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:09.115758 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:09.116826 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:09.117144 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:09.117144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:09.117145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:09.117192 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:09.117347 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:09.117347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:09.117348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:09.117487 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:09.117614 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.117671 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.117726 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:56:09.118600 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:09.120903 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.118900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.20.199.62:34247)
0925 14:56:09.118937 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.118939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.119022 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:56:09.119069 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.119070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.119097 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.119159 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:09.119224 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.119660 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.120341 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.120507 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.120554 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.120672 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:09.120756 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:09.120816 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.120845 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.120858 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:56:09.121202 3100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:09.122793 3100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:09.124958 3208 tablet_service.cc:1467] Processing CreateTablet for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 (DEFAULT_TABLE table=test-table [id=9599b247c4934c23b7f5457d438845ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:09.125144 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:09.126511 3180 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap starting.
I20260925 14:56:09.126845 3180 tablet_bootstrap.cc:654] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:09.127419 3180 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: No bootstrap required, opened a new log
I20260925 14:56:09.127470 3180 ts_tablet_manager.cc:1397] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:09.127565 3180 raft_consensus.cc:348] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.127629 3180 raft_consensus.cc:374] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:09.127665 3180 raft_consensus.cc:729] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b05c69a8e8a499d896ae3ab6ec951fb, State: Initialized, Role: FOLLOWER
I20260925 14:56:09.127708 3180 consensus_queue.cc:260] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.127796 3180 ts_tablet_manager.cc:1428] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.129012 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.124742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.20.199.3:34053)
0925 14:56:09.125165 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.125169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.125179 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:56:09.125210 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.125212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.125226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.125486 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:09.125544 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.126711 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.127926 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.128684 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.128752 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.128792 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.128900 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.128928 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":104,"threads_started":1}
I20260925 14:56:09.129014 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.124640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34053 (local address 127.0.0.1:55814)
0925 14:56:09.124690 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.124700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:09.124736 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:09.125337 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:09.125338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:09.125339 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:09.125377 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.125378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.126785 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.126786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.127732 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.127735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.128762 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.128762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.128823 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.128827 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:09.128889 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:56:09.128931 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:56:09.129472 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.124738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44161 (local address 127.0.0.1:35090)
0925 14:56:09.124823 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.124832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:09.124869 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:09.125312 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:09.125313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:09.125315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:09.125376 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.125378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.126692 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.126693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.127876 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.127879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.129018 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.129019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.129229 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.129235 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:56:09.129294 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:56:09.129336 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":58,"threads_started":1}
I20260925 14:56:09.129482 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.124936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.20.199.2:44161)
0925 14:56:09.125150 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.125155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.125167 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:56:09.125207 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.125210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.125226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.125312 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:09.125475 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.126607 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.127980 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.128936 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.128999 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.129053 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.129330 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.129377 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":112,"threads_started":1}
I20260925 14:56:09.129833 3492 tablet_service.cc:1467] Processing CreateTablet for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 (DEFAULT_TABLE table=test-table [id=9599b247c4934c23b7f5457d438845ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:09.130177 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:09.130089 3360 tablet_service.cc:1467] Processing CreateTablet for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 (DEFAULT_TABLE table=test-table [id=9599b247c4934c23b7f5457d438845ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:09.130380 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:09.133744 3567 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Bootstrap starting.
I20260925 14:56:09.134321 3567 tablet_bootstrap.cc:654] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:09.134752 3567 log.cc:824] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:09.135021 3568 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Bootstrap starting.
I20260925 14:56:09.135406 3567 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: No bootstrap required, opened a new log
I20260925 14:56:09.135460 3567 ts_tablet_manager.cc:1397] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:09.135596 3568 tablet_bootstrap.cc:654] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:09.136039 3568 log.cc:824] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:09.136870 3568 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: No bootstrap required, opened a new log
I20260925 14:56:09.136957 3568 ts_tablet_manager.cc:1397] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:09.137279 3567 raft_consensus.cc:348] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.137460 3567 raft_consensus.cc:374] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:09.137503 3567 raft_consensus.cc:729] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24900d3d26f949058067498ad7d86da1, State: Initialized, Role: FOLLOWER
I20260925 14:56:09.137651 3567 consensus_queue.cc:260] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.137965 3567 ts_tablet_manager.cc:1428] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:56:09.137990 3558 heartbeater.cc:503] Master 127.20.199.62:34247 was elected leader, sending a full tablet report...
I20260925 14:56:09.139022 3568 raft_consensus.cc:348] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.139186 3568 raft_consensus.cc:374] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:09.139232 3568 raft_consensus.cc:729] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36aeb434d33427a8e59c678f9e111bf, State: Initialized, Role: FOLLOWER
I20260925 14:56:09.139339 3568 consensus_queue.cc:260] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.139581 3568 ts_tablet_manager.cc:1428] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:56:09.139613 3426 heartbeater.cc:503] Master 127.20.199.62:34247 was elected leader, sending a full tablet report...
I20260925 14:56:09.177682 3572 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:09.177807 3572 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.178169 3572 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237)
I20260925 14:56:09.180938 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.178292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34053 (local address 127.20.199.2:37371)
0925 14:56:09.178341 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.178352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:09.178400 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:09.178624 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:09.178625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:09.178626 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:09.178701 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.178703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.179327 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.179328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.180056 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.180058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.180660 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.180661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.180745 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.180748 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:56:09.180824 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:56:09.180872 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 14:56:09.180938 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.178410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:37371 (local address 127.20.199.3:34053)
0925 14:56:09.178469 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.178491 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.178499 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:09.178529 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.178529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.178540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.178607 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:09.178777 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.179246 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.180149 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.180583 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.180664 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.180673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.180833 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.180868 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:56:09.181180 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.178460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:41223 (local address 127.20.199.1:43237)
0925 14:56:09.178529 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.178531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.178617 (+ 86us) server_negotiation.cc:407] Connection header received
0925 14:56:09.178697 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.178699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.178708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.178775 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:09.178938 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.179397 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.180373 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.180790 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.180884 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.180892 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.181080 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.181120 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:56:09.181181 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.178336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43237 (local address 127.20.199.2:41223)
0925 14:56:09.178545 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.178554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:09.178631 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:09.178793 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:09.178794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:09.178796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:09.178869 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.178871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.179483 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.179484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.180270 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:09.180272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.180889 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.180890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:09.180984 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.180988 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:09.181070 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:56:09.181107 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260925 14:56:09.181262 3512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712b5e02456a4aa3b35f2d2a4a44d5e1" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24900d3d26f949058067498ad7d86da1" is_pre_election: true
I20260925 14:56:09.181469 3512 raft_consensus.cc:2454] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 0.
I20260925 14:56:09.181484 3242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712b5e02456a4aa3b35f2d2a4a44d5e1" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" is_pre_election: true
I20260925 14:56:09.181630 3242 raft_consensus.cc:2454] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 0.
I20260925 14:56:09.181682 3316 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf; no voters:
I20260925 14:56:09.181824 3572 raft_consensus.cc:2781] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:09.181885 3572 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:09.181906 3572 raft_consensus.cc:3037] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:09.182639 3572 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.182770 3572 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 election: Requested vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237)
I20260925 14:56:09.182962 3242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712b5e02456a4aa3b35f2d2a4a44d5e1" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
I20260925 14:56:09.183015 3512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712b5e02456a4aa3b35f2d2a4a44d5e1" candidate_uuid: "a36aeb434d33427a8e59c678f9e111bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24900d3d26f949058067498ad7d86da1"
I20260925 14:56:09.183037 3242 raft_consensus.cc:3037] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:09.183100 3512 raft_consensus.cc:3037] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:09.183812 3512 raft_consensus.cc:2454] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 1.
I20260925 14:56:09.184020 3316 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf; no voters:
I20260925 14:56:09.184151 3242 raft_consensus.cc:2454] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36aeb434d33427a8e59c678f9e111bf in term 1.
I20260925 14:56:09.184167 3572 raft_consensus.cc:2781] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:09.184321 3572 raft_consensus.cc:686] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 1 LEADER]: Becoming Leader. State: Replica: a36aeb434d33427a8e59c678f9e111bf, State: Running, Role: LEADER
I20260925 14:56:09.184419 3572 consensus_queue.cc:237] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.185228 3099 catalog_manager.cc:5495] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf reported cstate change: term changed from 0 to 1, leader changed from <none> to a36aeb434d33427a8e59c678f9e111bf (127.20.199.2). New cstate: current_term: 1 leader_uuid: "a36aeb434d33427a8e59c678f9e111bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:09.201349 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.199198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.20.199.2:44161)
0925 14:56:09.199256 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:09.199258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:09.199265 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:56:09.199329 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:09.199330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:09.199347 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:09.199417 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:09.199522 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.199971 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.200656 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:09.200832 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:09.200883 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:09.201001 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:09.201105 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:09.201162 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:09.201207 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:56:09.201224 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:56:09.203087 3512 raft_consensus.cc:1261] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Refusing update from remote peer a36aeb434d33427a8e59c678f9e111bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:09.203100 3242 raft_consensus.cc:1261] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Refusing update from remote peer a36aeb434d33427a8e59c678f9e111bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:09.203459 3574 consensus_queue.cc:1035] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:09.203609 3572 consensus_queue.cc:1035] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:09.206869 3577 mvcc.cc:205] Tried to move back new op lower bound from 7333266101053247488 to 7333266100979990528. Current Snapshot: MvccSnapshot[applied={T|T < 7333266101053247488 or (T in {7333266101053247488})}]
I20260925 14:56:09.207901 3099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:09.207996 3099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:09.208753 3099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Sending DeleteTablet for 3 replicas of tablet 712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:09.209093 3492 tablet_service.cc:1514] Processing DeleteTablet for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:09 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:55814
I20260925 14:56:09.209141 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:09 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:45478
I20260925 14:56:09.209168 3360 tablet_service.cc:1514] Processing DeleteTablet for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:09 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260925 14:56:09.209215 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3138
I20260925 14:56:09.209285 3297 tablet_replica.cc:330] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: stopping tablet replica
I20260925 14:56:09.209321 3585 tablet_replica.cc:330] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: stopping tablet replica
I20260925 14:56:09.209342 3297 raft_consensus.cc:2229] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:09.209453 3297 raft_consensus.cc:2258] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:09.209776 3585 raft_consensus.cc:2229] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:09.209887 3585 raft_consensus.cc:2258] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:09.210158 3586 tablet_replica.cc:330] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: stopping tablet replica
I20260925 14:56:09.210261 3586 raft_consensus.cc:2229] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:09.210368 3586 raft_consensus.cc:2258] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:09.210752 3586 ts_tablet_manager.cc:1905] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:09.212975 3586 ts_tablet_manager.cc:1918] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:09.213053 3586 log.cc:1196] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:09.213367 3586 ts_tablet_manager.cc:1939] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P a36aeb434d33427a8e59c678f9e111bf: Deleting consensus metadata
I20260925 14:56:09.213579 3585 ts_tablet_manager.cc:1905] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:09.213781 3087 catalog_manager.cc:4841] TS a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161): tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 (table test-table [id=9599b247c4934c23b7f5457d438845ae]) successfully deleted
I20260925 14:56:09.215399 3585 ts_tablet_manager.cc:1918] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:09.215533 3585 log.cc:1196] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:09.215869 3585 ts_tablet_manager.cc:1939] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 24900d3d26f949058067498ad7d86da1: Deleting consensus metadata
W20260925 14:56:09.216121 3086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:43237 (error 108)
I20260925 14:56:09.216320 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3292
I20260925 14:56:09.216914 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.216468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:43237 (local address 127.0.0.1:45490)
0925 14:56:09.216844 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.216881 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
W20260925 14:56:09.217053 3086 catalog_manager.cc:4584] TS 5b05c69a8e8a499d896ae3ab6ec951fb (127.20.199.1:43237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 712b5e02456a4aa3b35f2d2a4a44d5e1: Network error: Client connection negotiation failed: client connection to 127.20.199.1:43237: connect: Connection refused (error 111)
I20260925 14:56:09.217654 3087 catalog_manager.cc:4841] TS 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): tablet 712b5e02456a4aa3b35f2d2a4a44d5e1 (table test-table [id=9599b247c4934c23b7f5457d438845ae]) successfully deleted
I20260925 14:56:09.222011 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3429
I20260925 14:56:09.226346 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3067
I20260925 14:56:09.231416 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34247
--webserver_interface=127.20.199.62
--webserver_port=44621
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:09.312135 3588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:09.312342 3588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:09.312373 3588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:09.313578 3588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:09.313622 3588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:09.313637 3588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:09.313649 3588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:09.313704 3588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:09.315181 3588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:09.315448 3588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:09.315691 3588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:09.320623 3594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.320663 3593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.320703 3588 server_base.cc:1034] running on GCE node
W20260925 14:56:09.320629 3596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.321067 3588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:09.321311 3588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:09.322482 3588 hybrid_clock.cc:648] HybridClock initialized: now 1790348169322479 us; error 35 us; skew 500 ppm
Sep 25 14:56:09 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:09.324858 3588 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:09.325232 3588 webserver.cc:460] Webserver started at http://127.20.199.62:44621/ using document root <none> and password file <none>
I20260925 14:56:09.325500 3588 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:09.325548 3588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:09.326825 3588 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:09.327443 3603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.327610 3588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:56:09.327711 3588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "a46d178deeb84b19bfd99e54355de8bf"
format_stamp: "Formatted at 2026-09-25 14:56:07 on dist-test-slave-8qt6"
server_key: "85be351a87865b0c2d9af4d8f2e93b47"
server_key_iv: "a61eca78a948ad8a99b69efae73e7f13"
server_key_version: "encryptionkey@0"
I20260925 14:56:09.328066 3588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:09.337317 3588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:09.339061 3588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:09.339216 3588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:09.343209 3588 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34247
I20260925 14:56:09.343240 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34247 every 8 connection(s)
I20260925 14:56:09.343609 3588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:09.345151 3656 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:56:09.345911 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap starting.
I20260925 14:56:09.346163 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3588
I20260925 14:56:09.346431 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:43237
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:34247
--webserver_port=33485
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:09.356539 3656 log.cc:824] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:09.359592 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:09.359946 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf: Bootstrap complete.
I20260925 14:56:09.363107 3656 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:09.363408 3656 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46d178deeb84b19bfd99e54355de8bf, State: Initialized, Role: FOLLOWER
I20260925 14:56:09.363564 3656 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:09.363626 3656 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:09.363667 3656 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:09.363711 3656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:56:09.364981 3656 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:09.365084 3656 leader_election.cc:302] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46d178deeb84b19bfd99e54355de8bf; no voters:
I20260925 14:56:09.365247 3656 leader_election.cc:288] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:56:09.365326 3660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:56:09.365509 3656 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:09.365517 3660 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [term 3 LEADER]: Becoming Leader. State: Replica: a46d178deeb84b19bfd99e54355de8bf, State: Running, Role: LEADER
I20260925 14:56:09.365664 3660 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } }
I20260925 14:56:09.365942 3660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46d178deeb84b19bfd99e54355de8bf. Latest consensus state: current_term: 3 leader_uuid: "a46d178deeb84b19bfd99e54355de8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } } }
I20260925 14:56:09.366051 3660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:09.365942 3661 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a46d178deeb84b19bfd99e54355de8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46d178deeb84b19bfd99e54355de8bf" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34247 } } }
I20260925 14:56:09.366204 3661 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46d178deeb84b19bfd99e54355de8bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:09.366264 3668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:09.367069 3668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:09.367352 3668 catalog_manager.cc:1259] Loaded cluster ID: 9386ed5c217c4bffa3c39631bc8f953c
I20260925 14:56:09.367399 3668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 14:56:09.367645 3668 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f04459069ad google::LogMessage::Fail() at ??:0
@ 0x7f044590988a google::LogMessage::SendToLog() at ??:0
@ 0x7f04459064de google::LogMessage::Flush() at ??:0
@ 0x7f0445907a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f044d8270f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f044d8308b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f04477d40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f04477c9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f044862d6db start_thread at ??:0
@ 0x7f044440461f clone at ??:0
W20260925 14:56:09.439906 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:09.440129 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:09.440167 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:09.441708 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:09.441764 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:09.441807 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:56:09.443428 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:43237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=33485
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:09.443689 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:09.443917 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:09.448899 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.448956 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.449064 3658 server_base.cc:1034] running on GCE node
W20260925 14:56:09.449079 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.449298 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:09.449620 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:09.450798 3658 hybrid_clock.cc:648] HybridClock initialized: now 1790348169450774 us; error 35 us; skew 500 ppm
Sep 25 14:56:09 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:09.453697 3658 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:56:09.454169 3658 webserver.cc:460] Webserver started at http://127.20.199.1:33485/ using document root <none> and password file <none>
I20260925 14:56:09.454401 3658 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:09.454469 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:09.455811 3658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:09.456501 3702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.456775 3658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:09.456894 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "5b05c69a8e8a499d896ae3ab6ec951fb"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "c2e62c939d1c87aed4b58bd0e94fd46f"
server_key_iv: "65e3595bb728bce9a830735a90cf2fb8"
server_key_version: "encryptionkey@0"
I20260925 14:56:09.457239 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:09.470537 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:09.472868 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:09.473042 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:09.473263 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:09.473795 3709 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:56:09.475608 3658 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:56:09.475669 3658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:56:09.475734 3658 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:56:09.476330 3658 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:56:09.476398 3658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:09.476457 3709 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap starting.
I20260925 14:56:09.481756 3658 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:43237
I20260925 14:56:09.482216 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:09.482574 3816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:43237 every 8 connection(s)
I20260925 14:56:09.487737 3709 log.cc:824] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:09.490047 3709 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:09.490325 3709 tablet_bootstrap.cc:492] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Bootstrap complete.
I20260925 14:56:09.491219 3709 ts_tablet_manager.cc:1397] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260925 14:56:09.492204 3709 raft_consensus.cc:348] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.492415 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3658
I20260925 14:56:09.492455 3709 raft_consensus.cc:729] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b05c69a8e8a499d896ae3ab6ec951fb, State: Initialized, Role: FOLLOWER
I20260925 14:56:09.492595 3709 consensus_queue.cc:260] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:09.492698 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:44161
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:34247
--webserver_port=33817
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:09.492906 3709 ts_tablet_manager.cc:1428] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260925 14:56:09.580116 3819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:56:09.482949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:46383)
0925 14:56:09.483121 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.483164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:09.483886 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:09.579902 (+ 96016us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
W20260925 14:56:09.580361 3817 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:34247 (0 consecutive failures): Network error: Failed to ping master at 127.20.199.62:34247: Client connection negotiation failed: client connection to 127.20.199.62:34247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:56:09.580711 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.580599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:52177)
0925 14:56:09.580653 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.580693 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:56:09.581058 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.580969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:41001)
0925 14:56:09.581012 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.581035 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:56:09.581164 3817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:56:09.585390 3822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:09.585644 3822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:09.585693 3822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:09.586918 3822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:09.586982 3822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:09.587025 3822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:56:09.588531 3822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:44161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=33817
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:09.588793 3822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:09.589027 3822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:09.593946 3830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.593951 3827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.593951 3828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.594283 3822 server_base.cc:1034] running on GCE node
I20260925 14:56:09.594486 3822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:09.594761 3822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:09.595916 3822 hybrid_clock.cc:648] HybridClock initialized: now 1790348169595922 us; error 61 us; skew 500 ppm
Sep 25 14:56:09 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:09.598428 3822 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:56:09.598799 3822 webserver.cc:460] Webserver started at http://127.20.199.2:33817/ using document root <none> and password file <none>
I20260925 14:56:09.599001 3822 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:09.599042 3822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:09.600279 3822 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:09.600813 3837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.600992 3822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:56:09.601085 3822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "a36aeb434d33427a8e59c678f9e111bf"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "8e60c47004b8507269417e5d805edc15"
server_key_iv: "22cdf0323b24ebc6b9de220cc6fae728"
server_key_version: "encryptionkey@0"
I20260925 14:56:09.601567 3822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:09.614172 3822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:09.616015 3822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:09.616175 3822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:09.616408 3822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:09.616715 3822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:09.616767 3822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.616809 3822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:09.616837 3822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.622947 3822 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:44161
I20260925 14:56:09.622989 3950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:44161 every 8 connection(s)
I20260925 14:56:09.623368 3822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:09.624526 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.624143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:45709)
0925 14:56:09.624302 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.624380 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
W20260925 14:56:09.624934 3951 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:34247 (0 consecutive failures): Network error: Failed to ping master at 127.20.199.62:34247: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
I20260925 14:56:09.625221 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.625131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:53351)
0925 14:56:09.625178 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.625203 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:56:09.625530 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.625471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:53133)
0925 14:56:09.625499 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.625517 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 14:56:09.625650 3951 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:56:09.628669 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3822
I20260925 14:56:09.628868 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:34053
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:34247
--webserver_port=34739
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:09.710901 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:09.711128 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:09.711154 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:09.712316 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:09.712359 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:09.712387 3954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:56:09.713912 3954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=34739
--tserver_master_addrs=127.20.199.62:34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:09.714157 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:09.714392 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:09.719547 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.719563 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:09.719534 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:09.719935 3954 server_base.cc:1034] running on GCE node
I20260925 14:56:09.720129 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:09.720358 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:09.721520 3954 hybrid_clock.cc:648] HybridClock initialized: now 1790348169721516 us; error 37 us; skew 500 ppm
Sep 25 14:56:09 dist-test-slave-8qt6 krb5kdc[2520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:09.723979 3954 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:56:09.724395 3954 webserver.cc:460] Webserver started at http://127.20.199.3:34739/ using document root <none> and password file <none>
I20260925 14:56:09.724610 3954 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:09.724675 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:09.726073 3954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:09.726625 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.726792 3954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:56:09.726910 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "24900d3d26f949058067498ad7d86da1"
format_stamp: "Formatted at 2026-09-25 14:56:08 on dist-test-slave-8qt6"
server_key: "15fb3d71cf499ee098104896299d046d"
server_key_iv: "b4c19aad436f6af0de183335e0588bbb"
server_key_version: "encryptionkey@0"
I20260925 14:56:09.727237 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:09.739123 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:09.741127 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:09.741281 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:09.741554 3954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:09.741894 3954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:09.742008 3954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.742074 3954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:09.742105 3954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:09.747982 3954 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:34053
I20260925 14:56:09.748042 4082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:34053 every 8 connection(s)
I20260925 14:56:09.748409 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:09.749684 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.749090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:49919)
0925 14:56:09.749239 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.749329 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260925 14:56:09.750110 4083 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:34247 (0 consecutive failures): Network error: Failed to ping master at 127.20.199.62:34247: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
I20260925 14:56:09.750396 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.750298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:59869)
0925 14:56:09.750346 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.750377 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:56:09.750797 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:09.750711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:50063)
0925 14:56:09.750759 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:09.750782 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 14:56:09.750967 4083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:56:09.753463 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 3954
I20260925 14:56:10.581818 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:10.581578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:57745)
0925 14:56:10.581709 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:10.581742 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 14:56:10.626298 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:10.626039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:40455)
0925 14:56:10.626202 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:10.626240 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260925 14:56:10.751549 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:10.751326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:33659)
0925 14:56:10.751453 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:10.751492 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260925 14:56:11.000865 4090 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:11.001025 4090 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:11.001439 4090 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
Sep 25 14:56:11 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.2@KRBTEST.COM
Sep 25 14:56:11 dist-test-slave-8qt6 krb5kdc[2520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.3@KRBTEST.COM
I20260925 14:56:11.011869 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.001942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38689 (local address 127.20.199.2:44161)
0925 14:56:11.002087 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:11.002097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:11.002131 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:56:11.002537 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:11.002539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:11.002576 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:11.002717 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:11.003382 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.004314 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.005390 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.005815 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.005947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:11.008296 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:11.008301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:11.008302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:11.008313 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:11.010029 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:11.010519 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:11.010522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:11.010522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:11.010621 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:11.010898 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:11.010899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:11.010899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:11.011209 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:11.011268 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:11.011592 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:56:11.011721 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260925 14:56:11.011863 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.001723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:44161 (local address 127.20.199.1:38689)
0925 14:56:11.001910 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:11.001920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:11.001980 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:11.002707 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:11.002708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:11.003067 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:11.003258 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:11.003264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.004419 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.004420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:11.005306 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:11.005309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.005925 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.005926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:11.006033 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:11.006605 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:11.006613 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:11.008157 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:11.010114 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:11.010121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:11.010123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:11.010432 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:11.010692 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:11.010692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:11.010693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:11.010825 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:11.011277 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:11.011283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:11.011419 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:56:11.011527 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:56:11.011724 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":28,"thread_start_us":66,"threads_started":1}
I20260925 14:56:11.011968 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.001600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34053 (local address 127.20.199.1:41439)
0925 14:56:11.001665 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:11.001677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:11.001753 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:11.002667 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:11.002670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:11.003065 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:11.003258 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:11.003264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.004452 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.004453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:11.005307 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:11.005310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.005921 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.005922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:11.006033 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:11.006663 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:11.006669 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:11.008482 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:11.010512 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:11.010513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:11.010514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:11.010722 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:11.010999 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:11.011000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:11.011000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:11.011070 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:11.011484 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:11.011484 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:11.011544 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 14:56:11.011617 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:56:11.011805 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":141}
I20260925 14:56:11.012522 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.001749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:41439 (local address 127.20.199.3:34053)
0925 14:56:11.001864 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:11.001891 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:11.001930 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:56:11.002427 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:11.002429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:11.002474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:11.002625 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:11.003385 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.004361 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.005432 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:11.005815 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:11.005946 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:11.008578 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:11.008582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:11.008582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:11.008591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:11.010428 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:11.010806 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:11.010809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:11.010809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:11.010926 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:11.011134 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:11.011135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:11.011135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:11.011407 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:11.011491 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:11.011696 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:56:11.011813 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":52,"threads_started":1}
W20260925 14:56:11.012625 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
W20260925 14:56:11.013124 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:11.013178 3706 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb; no voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf
I20260925 14:56:11.013288 4090 raft_consensus.cc:2726] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:56:11.582477 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.582286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:47279)
0925 14:56:11.582401 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:11.582431 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260925 14:56:11.626965 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.626766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:42861)
0925 14:56:11.626891 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:11.626924 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260925 14:56:11.752231 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:11.752034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:42881)
0925 14:56:11.752151 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:11.752189 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260925 14:56:12.522913 4097 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:12.523005 4097 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:12.523185 4097 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
W20260925 14:56:12.523582 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
W20260925 14:56:12.523690 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:12.523725 3706 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb; no voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf
I20260925 14:56:12.523816 4097 raft_consensus.cc:2726] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:56:12.583254 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:12.583031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:34409)
0925 14:56:12.583168 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:12.583207 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260925 14:56:12.627660 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:12.627455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:37465)
0925 14:56:12.627576 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:12.627618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 14:56:12.752944 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:12.752733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:60567)
0925 14:56:12.752863 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:12.752903 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260925 14:56:13.583990 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:13.583772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:36439)
0925 14:56:13.583902 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:13.583943 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 14:56:13.628410 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:13.628181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:60025)
0925 14:56:13.628319 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:13.628360 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260925 14:56:13.753748 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:13.753500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:59455)
0925 14:56:13.753660 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:13.753706 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 14:56:14.584715 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:14.584516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:44833)
0925 14:56:14.584634 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:14.584670 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260925 14:56:14.629092 4105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:14.628888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:52949)
0925 14:56:14.629017 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:14.629054 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260925 14:56:14.754451 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:14.754208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:39177)
0925 14:56:14.754347 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:14.754410 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260925 14:56:15.585381 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:15.585178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:57635)
0925 14:56:15.585295 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:15.585329 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260925 14:56:15.629935 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:15.629633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:42439)
0925 14:56:15.629835 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:15.629893 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":155,"threads_started":1}
I20260925 14:56:15.736109 4109 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:15.736197 4109 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:15.736321 4109 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
W20260925 14:56:15.736735 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
W20260925 14:56:15.736822 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:15.736855 3706 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb; no voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf
I20260925 14:56:15.736937 4109 raft_consensus.cc:2726] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:56:15.755079 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:15.754869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:49949)
0925 14:56:15.755004 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:15.755039 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 14:56:16.586076 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:16.585876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:55825)
0925 14:56:16.585996 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:16.586032 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260925 14:56:16.630738 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:16.630517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:40685)
0925 14:56:16.630664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:16.630700 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260925 14:56:16.755744 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:16.755526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:35617)
0925 14:56:16.755662 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:16.755702 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260925 14:56:17.586930 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:17.586688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:46683)
0925 14:56:17.586849 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:17.586883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260925 14:56:17.631335 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:17.631146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:51611)
0925 14:56:17.631258 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:17.631292 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260925 14:56:17.756438 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:17.756225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:58317)
0925 14:56:17.756346 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:17.756396 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 14:56:18.587621 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:18.587409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:38969)
0925 14:56:18.587538 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:18.587577 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260925 14:56:18.632005 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:18.631806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:42999)
0925 14:56:18.631926 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:18.631963 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260925 14:56:18.757190 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:18.756997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:48499)
0925 14:56:18.757113 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:18.757148 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260925 14:56:19.588451 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:19.588236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:47779)
0925 14:56:19.588365 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:19.588407 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260925 14:56:19.632623 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:19.632433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:35073)
0925 14:56:19.632547 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:19.632579 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260925 14:56:19.757808 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:19.757614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:43955)
0925 14:56:19.757729 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:19.757765 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260925 14:56:20.589252 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:20.589017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:55357)
0925 14:56:20.589166 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:20.589207 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260925 14:56:20.633325 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:20.633119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:44341)
0925 14:56:20.633241 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:20.633282 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260925 14:56:20.640515 4125 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:20.640625 4125 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:20.640854 4125 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
W20260925 14:56:20.641336 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
W20260925 14:56:20.641469 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:20.641505 3706 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb; no voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf
I20260925 14:56:20.641604 4125 raft_consensus.cc:2726] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:56:20.758478 4126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:20.758247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:57837)
0925 14:56:20.758383 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:20.758420 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260925 14:56:21.589942 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:21.589732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:57229)
0925 14:56:21.589857 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:21.589896 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260925 14:56:21.634131 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:21.633917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:36229)
0925 14:56:21.634047 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:21.634087 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260925 14:56:21.759264 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:21.758947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:57345)
0925 14:56:21.759180 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:21.759220 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":73,"threads_started":1}
I20260925 14:56:22.590713 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:22.590492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:44265)
0925 14:56:22.590622 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:22.590668 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260925 14:56:22.634822 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:22.634564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:43527)
0925 14:56:22.634728 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:22.634773 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260925 14:56:22.759963 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:22.759746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:38687)
0925 14:56:22.759876 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:22.759919 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 14:56:22.851552 4133 raft_consensus.cc:482] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:22.851648 4133 raft_consensus.cc:504] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24900d3d26f949058067498ad7d86da1" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34053 } } peers { permanent_uuid: "a36aeb434d33427a8e59c678f9e111bf" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 44161 } } peers { permanent_uuid: "5b05c69a8e8a499d896ae3ab6ec951fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 43237 } }
I20260925 14:56:22.851856 4133 leader_election.cc:288] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053), a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161)
W20260925 14:56:22.852330 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24900d3d26f949058067498ad7d86da1 (127.20.199.3:34053): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
W20260925 14:56:22.852438 3706 leader_election.cc:341] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a36aeb434d33427a8e59c678f9e111bf (127.20.199.2:44161): Not found: Tablet not found: 712b5e02456a4aa3b35f2d2a4a44d5e1
I20260925 14:56:22.852474 3706 leader_election.cc:302] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b05c69a8e8a499d896ae3ab6ec951fb; no voters: 24900d3d26f949058067498ad7d86da1, a36aeb434d33427a8e59c678f9e111bf
I20260925 14:56:22.852571 4133 raft_consensus.cc:2726] T 712b5e02456a4aa3b35f2d2a4a44d5e1 P 5b05c69a8e8a499d896ae3ab6ec951fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:56:23.591529 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:23.591285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:32935)
0925 14:56:23.591437 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:23.591482 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260925 14:56:23.635578 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:23.635363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:59309)
0925 14:56:23.635497 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:23.635534 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260925 14:56:23.760691 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:23.760445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:49027)
0925 14:56:23.760599 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:23.760645 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260925 14:56:24.592388 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:24.592119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.1:36315)
0925 14:56:24.592284 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:24.592343 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260925 14:56:24.636317 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:24.636095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.2:48795)
0925 14:56:24.636231 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:24.636272 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260925 14:56:24.761473 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:24.761208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:34247 (local address 127.20.199.3:59647)
0925 14:56:24.761326 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:24.761362 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:34247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260925 14:56:24.763895 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3658
I20260925 14:56:24.768716 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3822
I20260925 14:56:24.773073 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 3954
2026-09-25T14:56:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18207 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:24 dist-test-slave-8qt6 krb5kdc[4142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:24 dist-test-slave-8qt6 krb5kdc[4142](info): set up 2 sockets
Sep 25 14:56:24 dist-test-slave-8qt6 krb5kdc[4142](info): commencing operation
krb5kdc: starting...
W20260925 14:56:25.821585 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:56:25 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:56:25.942557 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45219
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:26.035573 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:26.035909 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:26.035949 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:26.038216 4155 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:56:26.038300 4155 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:56:26.038327 4155 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:56:26.038375 4155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:26.038424 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:26.038460 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:26.038517 4155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:26.038550 4155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:26.040448 4155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:26.040781 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:26.041041 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:26.046818 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.047019 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:26.047358 4155 server_base.cc:1034] running on GCE node
W20260925 14:56:26.048409 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:26.050151 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:26.050485 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:26.051638 4155 hybrid_clock.cc:648] HybridClock initialized: now 1790348186051603 us; error 46 us; skew 500 ppm
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:26.055452 4155 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:26.056121 4155 webserver.cc:460] Webserver started at http://127.20.199.62:43871/ using document root <none> and password file <none>
I20260925 14:56:26.056401 4155 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:26.056453 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:26.056608 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:26.057921 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "b55b65bb22ce4b67bc75bb410820bb0e"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "3a48f6658fc5a18f3bbeddd930d3d513"
server_key_iv: "430f9097b3d4e5968fc905aff86be80c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.058256 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "b55b65bb22ce4b67bc75bb410820bb0e"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "3a48f6658fc5a18f3bbeddd930d3d513"
server_key_iv: "430f9097b3d4e5968fc905aff86be80c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.059660 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:56:26.060474 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.060683 4155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:26.060783 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "b55b65bb22ce4b67bc75bb410820bb0e"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "3a48f6658fc5a18f3bbeddd930d3d513"
server_key_iv: "430f9097b3d4e5968fc905aff86be80c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.060911 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:26.072188 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:26.074203 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:26.074371 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:26.078400 4155 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:45219
I20260925 14:56:26.078438 4222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:45219 every 8 connection(s)
I20260925 14:56:26.078815 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:26.079648 4223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:26.082021 4223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap starting.
I20260925 14:56:26.082602 4223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:26.082904 4223 log.cc:824] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:26.083582 4223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: No bootstrap required, opened a new log
I20260925 14:56:26.085213 4223 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:26.085435 4223 raft_consensus.cc:374] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:26.085482 4223 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55b65bb22ce4b67bc75bb410820bb0e, State: Initialized, Role: FOLLOWER
I20260925 14:56:26.085633 4223 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:26.085716 4223 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:26.085755 4223 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:26.085807 4223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:26.086335 4223 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:26.086449 4223 leader_election.cc:302] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55b65bb22ce4b67bc75bb410820bb0e; no voters:
I20260925 14:56:26.086634 4223 leader_election.cc:288] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:26.086683 4226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:26.086861 4226 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 LEADER]: Becoming Leader. State: Replica: b55b65bb22ce4b67bc75bb410820bb0e, State: Running, Role: LEADER
I20260925 14:56:26.086910 4223 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:26.086944 4226 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:26.087618 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } } }
I20260925 14:56:26.087648 4228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55b65bb22ce4b67bc75bb410820bb0e. Latest consensus state: current_term: 1 leader_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } } }
I20260925 14:56:26.087716 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:26.087889 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4155
I20260925 14:56:26.088001 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:56:26.088246 21276 external_mini_cluster.cc:1428] Setting key 1062dc4fa5ef8ba51194f7f31af9ff39
I20260925 14:56:26.087723 4228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:26.088315 4235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:26.089418 4235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:26.094730 4235 catalog_manager.cc:1347] Generated new cluster ID: bab5139583264ea3929ed3756b28af8d
I20260925 14:56:26.094800 4235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:26.109146 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.088758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.20.199.62:45219)
0925 14:56:26.089085 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.089094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.089139 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:56:26.089254 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.089260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.089307 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.089491 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:26.090342 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.091837 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.093002 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.093743 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.094191 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.105035 (+ 10844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:26.105040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:26.105040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:26.105055 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:26.107128 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.107523 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.107526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.107527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.107638 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.107813 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.107813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.107813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.108136 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:26.108252 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.108735 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.108901 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":76,"threads_started":1}
I20260925 14:56:26.115499 4235 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:26.116022 4235 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:26.135254 4235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Generated new TSK 0
I20260925 14:56:26.135485 4235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:56:26.166436 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:45219
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:26.271503 4249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:26.271735 4249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:26.271776 4249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:26.273023 4249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:26.273089 4249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:26.273132 4249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:56:26.274842 4249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:26.275121 4249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:26.275372 4249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:26.280339 4254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.280375 4257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.280375 4255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:26.280537 4249 server_base.cc:1034] running on GCE node
I20260925 14:56:26.280766 4249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:26.281008 4249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:26.282166 4249 hybrid_clock.cc:648] HybridClock initialized: now 1790348186282123 us; error 54 us; skew 500 ppm
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:26.284792 4249 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:56:26.285266 4249 webserver.cc:460] Webserver started at http://127.20.199.1:43771/ using document root <none> and password file <none>
I20260925 14:56:26.285670 4249 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:26.285742 4249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:26.285877 4249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:26.287048 4249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "e0f3b66185a800fa78bd116aea32f237"
server_key_iv: "76d8267c50d6657fca0ee733d5b83e61"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.287420 4249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "e0f3b66185a800fa78bd116aea32f237"
server_key_iv: "76d8267c50d6657fca0ee733d5b83e61"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.288828 4249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:56:26.289645 4264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.289813 4249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:26.289918 4249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "e0f3b66185a800fa78bd116aea32f237"
server_key_iv: "76d8267c50d6657fca0ee733d5b83e61"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.290006 4249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:26.301505 4249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:26.303853 4249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:26.304018 4249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:26.304260 4249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:26.304574 4249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:26.304626 4249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.304669 4249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:26.304694 4249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.311102 4249 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:34943
I20260925 14:56:26.311142 4377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:34943 every 8 connection(s)
I20260925 14:56:26.311510 4249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:26.311995 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4249
I20260925 14:56:26.312119 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:26.312337 21276 external_mini_cluster.cc:1428] Setting key cad99c4baf822ad052973b40c018d81d
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
I20260925 14:56:26.322201 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.312924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:51937 (local address 127.20.199.62:45219)
0925 14:56:26.312973 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.312975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.313255 (+ 280us) server_negotiation.cc:407] Connection header received
0925 14:56:26.313856 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.313857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.313872 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.313928 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:26.314907 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.315425 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.316446 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.316655 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.316732 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.319186 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:26.319188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:26.319188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:26.319198 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:26.320734 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.321178 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.321179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.321179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.321240 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.321550 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.321551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.321551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.321668 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:26.321747 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.322025 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.322109 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:56:26.322396 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.312933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:51937)
0925 14:56:26.313111 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.313158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.313748 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.314002 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.314006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.314587 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:26.314785 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.314790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.315531 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.315532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.316351 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.316353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.316740 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.316741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.316882 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.317517 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:26.317526 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:26.319086 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:26.320868 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:26.320876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:26.320878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:26.321098 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:26.321309 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:26.321310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:26.321311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:26.321481 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:26.321738 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:26.321744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:26.321889 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.321992 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.322148 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 14:56:26.322873 4378 heartbeater.cc:347] Connected to a master server at 127.20.199.62:45219
I20260925 14:56:26.322975 4378 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
I20260925 14:56:26.323160 4378 heartbeater.cc:511] Master 127.20.199.62:45219 requested a full tablet report, sending...
I20260925 14:56:26.323738 4187 ts_manager.cc:194] Registered new tserver with Master: dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:56:26.324659 4187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:51937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:56:26.341111 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:45219
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:26.421432 4383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:26.421684 4383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:26.421723 4383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:26.422919 4383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:26.422996 4383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:26.423040 4383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:56:26.424501 4383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:26.424778 4383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:26.425020 4383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:26.430006 4389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.430056 4388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.430018 4391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:26.430054 4383 server_base.cc:1034] running on GCE node
I20260925 14:56:26.430368 4383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:26.430606 4383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:26.431766 4383 hybrid_clock.cc:648] HybridClock initialized: now 1790348186431746 us; error 35 us; skew 500 ppm
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:26.434373 4383 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:56:26.434780 4383 webserver.cc:460] Webserver started at http://127.20.199.2:42935/ using document root <none> and password file <none>
I20260925 14:56:26.435016 4383 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:26.435084 4383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:26.435235 4383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:26.436290 4383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "5b5c325117754e3482bd9734bf13076b"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "9d0e533bcc62fcb6775fba0931d5b719"
server_key_iv: "11324f682472247f7b9756212bfd9e3c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.436642 4383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "5b5c325117754e3482bd9734bf13076b"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "9d0e533bcc62fcb6775fba0931d5b719"
server_key_iv: "11324f682472247f7b9756212bfd9e3c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.438081 4383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:56:26.438856 4398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.439033 4383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:26.439134 4383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "5b5c325117754e3482bd9734bf13076b"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "9d0e533bcc62fcb6775fba0931d5b719"
server_key_iv: "11324f682472247f7b9756212bfd9e3c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.439208 4383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:26.450799 4383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:26.453183 4383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:26.453338 4383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:26.453635 4383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:26.453994 4383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:26.454046 4383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.454082 4383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:26.454109 4383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.459693 4383 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:38635
I20260925 14:56:26.459743 4511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:38635 every 8 connection(s)
I20260925 14:56:26.460083 4383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:26.466068 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4383
I20260925 14:56:26.466146 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:26.466369 21276 external_mini_cluster.cc:1428] Setting key b7247911e648d69c5d7590231bff9d33
I20260925 14:56:26.469638 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.460785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:44599 (local address 127.20.199.62:45219)
0925 14:56:26.460841 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.460843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.461029 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:56:26.461572 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.461572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.461583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.461643 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:26.462570 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.463061 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.464147 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.464328 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.464407 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.466834 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:26.466835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:26.466836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:26.466843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:26.468188 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.468611 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.468612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.468612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.468673 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.468939 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.468940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.468940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.469061 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:26.469134 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.469461 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.469549 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:56:26.469838 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.460739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:44599)
0925 14:56:26.460898 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.460940 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.461474 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.461741 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.461744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.462303 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:26.462492 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.462497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.463145 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.463146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.464068 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.464070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.464405 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.464406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.464529 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.465080 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:26.465089 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:26.466719 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:26.468285 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:26.468292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:26.468294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:26.468512 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:26.468739 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:26.468740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:26.468740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:26.468861 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:26.469136 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:26.469143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:26.469281 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.469389 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.469585 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260925 14:56:26.470283 4512 heartbeater.cc:347] Connected to a master server at 127.20.199.62:45219
I20260925 14:56:26.470381 4512 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:26.470571 4512 heartbeater.cc:511] Master 127.20.199.62:45219 requested a full tablet report, sending...
I20260925 14:56:26.470954 4187 ts_manager.cc:194] Registered new tserver with Master: 5b5c325117754e3482bd9734bf13076b (127.20.199.2:38635)
I20260925 14:56:26.471501 4187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:44599
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:56:26.490736 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:45219
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:26.571269 4517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:26.571478 4517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:26.571501 4517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:26.572590 4517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:26.572624 4517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:26.572649 4517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:56:26.574091 4517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:26.574280 4517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:26.574457 4517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:26.579452 4522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.579438 4525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:26.579438 4523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:26.579730 4517 server_base.cc:1034] running on GCE node
I20260925 14:56:26.579986 4517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:26.580276 4517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:26.581463 4517 hybrid_clock.cc:648] HybridClock initialized: now 1790348186581462 us; error 38 us; skew 500 ppm
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:26.583825 4517 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:56:26.584244 4517 webserver.cc:460] Webserver started at http://127.20.199.3:42305/ using document root <none> and password file <none>
I20260925 14:56:26.584472 4517 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:26.584539 4517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:26.584693 4517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:26.585803 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "2b5dd4eab3171166941c5ca52c675e93"
server_key_iv: "4b339df5875e3a750c9eedbb8350549c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.586196 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "2b5dd4eab3171166941c5ca52c675e93"
server_key_iv: "4b339df5875e3a750c9eedbb8350549c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.588205 4517 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:26.589224 4532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.589555 4517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:26.589668 4517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "2b5dd4eab3171166941c5ca52c675e93"
server_key_iv: "4b339df5875e3a750c9eedbb8350549c"
server_key_version: "encryptionkey@0"
I20260925 14:56:26.589793 4517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:26.605392 4517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:26.607539 4517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:26.607729 4517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:26.608001 4517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:26.608335 4517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:26.608393 4517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.608434 4517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:26.608464 4517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:26.614221 4517 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:33191
I20260925 14:56:26.614333 4645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:33191 every 8 connection(s)
I20260925 14:56:26.614710 4517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:26.615753 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4517
I20260925 14:56:26.615859 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:26.616073 21276 external_mini_cluster.cc:1428] Setting key 0177fec0993d3b4cbe36768f064d74b9
Sep 25 14:56:26 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:26.624729 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.615906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:49821 (local address 127.20.199.62:45219)
0925 14:56:26.615962 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.615964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.616311 (+ 347us) server_negotiation.cc:407] Connection header received
0925 14:56:26.617012 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.617013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.617025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.617070 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:26.618089 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.618596 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.619599 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.619817 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.619902 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.622069 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:26.622071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:26.622071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:26.622078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:26.623360 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.623763 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.623764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.623765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.623829 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.624067 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.624068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.624068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.624186 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:26.624250 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.624561 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.624651 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:56:26.624797 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.615900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:49821)
0925 14:56:26.616093 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.616182 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.616900 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.617139 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.617143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.617807 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:26.617993 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.617998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.618691 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.618692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.619505 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.619507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.619913 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.619914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.620041 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.620486 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:26.620494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:26.621982 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:26.623446 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:26.623451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:26.623452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:26.623668 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:26.623907 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:26.623908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:26.623908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:26.624005 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:26.624267 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:26.624273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:26.624390 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.624505 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.624632 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260925 14:56:26.625329 4646 heartbeater.cc:347] Connected to a master server at 127.20.199.62:45219
I20260925 14:56:26.625458 4646 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:26.625655 4646 heartbeater.cc:511] Master 127.20.199.62:45219 requested a full tablet report, sending...
I20260925 14:56:26.626179 4187 ts_manager.cc:194] Registered new tserver with Master: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191)
I20260925 14:56:26.626536 4187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:49821
I20260925 14:56:26.628043 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:26.635532 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.629447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.20.199.62:45219)
0925 14:56:26.629498 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.629500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.629511 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:56:26.629583 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.629584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.629594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.629644 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:26.630207 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.630671 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.631403 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.631597 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.631670 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.632860 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:26.632861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:26.632862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:26.632869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:26.634323 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.634688 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.634689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.634689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.634753 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:26.634952 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:26.634953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:26.634953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:26.635111 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:26.635219 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.635403 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.635469 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:56:26.636972 4187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:26.638631 4187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:26.645370 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.640979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33191 (local address 127.0.0.1:56298)
0925 14:56:26.641345 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.641388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.641487 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.641912 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.641913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.641915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:26.641963 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.641965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.643278 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.643279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.644224 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.644226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.645146 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.645146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.645213 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.645215 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.645259 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.645300 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260925 14:56:26.645558 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.641044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.0.0.1:58578)
0925 14:56:26.641344 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.641388 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.641486 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.641826 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.641828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.641830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:26.641895 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.641898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.642965 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.642966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.644336 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.644338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.645263 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.645264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.645336 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.645340 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.645435 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.645482 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":92,"threads_started":1}
I20260925 14:56:26.645555 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.641057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.20.199.3:33191)
0925 14:56:26.641626 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.641630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.641640 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:56:26.641680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.641682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.641698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.641815 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:26.642040 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.643161 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.644319 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.645037 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.645110 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.645168 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.645290 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.645336 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":70,"threads_started":1}
I20260925 14:56:26.645823 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.641148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.20.199.1:34943)
0925 14:56:26.641338 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.641343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.641592 (+ 249us) server_negotiation.cc:407] Connection header received
0925 14:56:26.641639 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.641641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.641659 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.641731 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:26.641977 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.642865 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.644422 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.645124 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.645183 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.645228 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.645672 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.645723 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260925 14:56:26.645886 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.641189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:38635 (local address 127.0.0.1:51870)
0925 14:56:26.641551 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.641561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.641609 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.641907 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.641908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.641910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:26.641953 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.641954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.643260 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.643261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.644373 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.644374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.645038 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.645039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.645113 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.645127 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.645193 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.645260 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":31,"thread_start_us":98,"threads_started":1}
I20260925 14:56:26.646039 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.641307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.20.199.2:38635)
0925 14:56:26.641496 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.641500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.641663 (+ 163us) server_negotiation.cc:407] Connection header received
0925 14:56:26.641705 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.641708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.641723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.641832 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:26.642059 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.643179 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.644464 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.644931 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.645034 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.645098 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.645341 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.645391 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":133,"threads_started":1}
I20260925 14:56:26.646401 4312 tablet_service.cc:1467] Processing CreateTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca (DEFAULT_TABLE table=test-table [id=dd3721c67a9a42eab39ac52ca6a1fb1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:26.646402 4580 tablet_service.cc:1467] Processing CreateTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca (DEFAULT_TABLE table=test-table [id=dd3721c67a9a42eab39ac52ca6a1fb1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:26.646584 4446 tablet_service.cc:1467] Processing CreateTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca (DEFAULT_TABLE table=test-table [id=dd3721c67a9a42eab39ac52ca6a1fb1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:26.646734 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:26.646734 4312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:26.646826 4446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:26.649758 4663 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Bootstrap starting.
I20260925 14:56:26.650142 4665 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap starting.
I20260925 14:56:26.650480 4663 tablet_bootstrap.cc:654] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:26.650710 4665 tablet_bootstrap.cc:654] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:26.651064 4663 log.cc:824] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:26.651212 4665 log.cc:824] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:26.651456 4664 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Bootstrap starting.
I20260925 14:56:26.651853 4663 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: No bootstrap required, opened a new log
I20260925 14:56:26.651939 4663 ts_tablet_manager.cc:1397] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:26.652021 4665 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: No bootstrap required, opened a new log
I20260925 14:56:26.652086 4665 ts_tablet_manager.cc:1397] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:26.652213 4664 tablet_bootstrap.cc:654] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:26.652755 4664 log.cc:824] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:26.653644 4664 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: No bootstrap required, opened a new log
I20260925 14:56:26.653774 4664 ts_tablet_manager.cc:1397] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:26.653770 4663 raft_consensus.cc:348] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.653968 4663 raft_consensus.cc:374] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:26.654011 4663 raft_consensus.cc:729] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5c325117754e3482bd9734bf13076b, State: Initialized, Role: FOLLOWER
I20260925 14:56:26.654160 4663 consensus_queue.cc:260] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.654458 4663 ts_tablet_manager.cc:1428] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:56:26.654400 4665 raft_consensus.cc:348] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.654603 4512 heartbeater.cc:503] Master 127.20.199.62:45219 was elected leader, sending a full tablet report...
I20260925 14:56:26.654608 4665 raft_consensus.cc:374] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:26.654657 4665 raft_consensus.cc:729] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba34c3b84cc431aa6a5fa4d6428e848, State: Initialized, Role: FOLLOWER
I20260925 14:56:26.654775 4665 consensus_queue.cc:260] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.655066 4378 heartbeater.cc:503] Master 127.20.199.62:45219 was elected leader, sending a full tablet report...
I20260925 14:56:26.655067 4665 ts_tablet_manager.cc:1428] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:26.656549 4664 raft_consensus.cc:348] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.656687 4664 raft_consensus.cc:374] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:26.656718 4664 raft_consensus.cc:729] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6, State: Initialized, Role: FOLLOWER
I20260925 14:56:26.656826 4664 consensus_queue.cc:260] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.657131 4664 ts_tablet_manager.cc:1428] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:26.657145 4646 heartbeater.cc:503] Master 127.20.199.62:45219 was elected leader, sending a full tablet report...
I20260925 14:56:26.691497 4669 raft_consensus.cc:482] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:26.691636 4669 raft_consensus.cc:504] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.692474 4669 leader_election.cc:288] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191), dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:56:26.695266 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.692405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:53677 (local address 127.20.199.1:34943)
0925 14:56:26.692463 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.692465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.692589 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:56:26.692677 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.692678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.692691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.692753 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:26.692918 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.693578 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.694527 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.694920 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.695005 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.695014 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.695168 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.695202 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:56:26.695274 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.692319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.20.199.2:53677)
0925 14:56:26.692539 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.692547 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.692613 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.692770 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.692771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.692772 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:26.692867 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.692868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.693661 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.693662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.694435 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.694437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.695005 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.695006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.695075 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.695080 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.695151 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.695193 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
I20260925 14:56:26.695501 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.692405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:38279 (local address 127.20.199.3:33191)
0925 14:56:26.692462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.692465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.692495 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:56:26.692526 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.692527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.692538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.692595 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:26.692774 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.693642 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.694649 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.695066 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.695160 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.695169 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.695375 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.695442 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:56:26.695508 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.692304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33191 (local address 127.20.199.2:38279)
0925 14:56:26.692363 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.692375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:26.692437 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:26.692609 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:26.692610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:26.692612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:26.692683 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.692685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.693730 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.693731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.694558 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:26.694560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.695161 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.695162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:26.695248 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.695251 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:56:26.695360 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:56:26.695407 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:56:26.695796 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50c9b4d45b14fcb81e4c8dd25a7a0ca" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" is_pre_election: true
I20260925 14:56:26.695916 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50c9b4d45b14fcb81e4c8dd25a7a0ca" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" is_pre_election: true
I20260925 14:56:26.695969 4332 raft_consensus.cc:2454] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 0.
I20260925 14:56:26.696074 4600 raft_consensus.cc:2454] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 0.
I20260925 14:56:26.696182 4399 leader_election.cc:302] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5c325117754e3482bd9734bf13076b, dba34c3b84cc431aa6a5fa4d6428e848; no voters:
I20260925 14:56:26.696318 4669 raft_consensus.cc:2781] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:26.696398 4669 raft_consensus.cc:482] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:26.696440 4669 raft_consensus.cc:3037] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:26.697212 4669 raft_consensus.cc:504] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.697376 4669 leader_election.cc:288] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 election: Requested vote from peers 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191), dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:56:26.697536 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50c9b4d45b14fcb81e4c8dd25a7a0ca" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
I20260925 14:56:26.697563 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50c9b4d45b14fcb81e4c8dd25a7a0ca" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
I20260925 14:56:26.697613 4600 raft_consensus.cc:3037] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:26.697647 4332 raft_consensus.cc:3037] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:26.698557 4332 raft_consensus.cc:2454] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 1.
I20260925 14:56:26.698555 4600 raft_consensus.cc:2454] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 1.
I20260925 14:56:26.698714 4399 leader_election.cc:302] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5c325117754e3482bd9734bf13076b, dba34c3b84cc431aa6a5fa4d6428e848; no voters:
I20260925 14:56:26.698828 4669 raft_consensus.cc:2781] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:26.698978 4669 raft_consensus.cc:686] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 1 LEADER]: Becoming Leader. State: Replica: 5b5c325117754e3482bd9734bf13076b, State: Running, Role: LEADER
I20260925 14:56:26.699079 4669 consensus_queue.cc:237] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:26.699889 4186 catalog_manager.cc:5495] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5c325117754e3482bd9734bf13076b (127.20.199.2). New cstate: current_term: 1 leader_uuid: "5b5c325117754e3482bd9734bf13076b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } health_report { overall_health: UNKNOWN } } }
W20260925 14:56:26.710611 4513 tablet.cc:2369] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:26.718232 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.715841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.20.199.2:38635)
0925 14:56:26.715895 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:26.715900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:26.715915 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:56:26.715974 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:26.715975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:26.715993 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:26.716060 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:26.716176 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.716732 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.717519 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:26.717748 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:26.717808 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:26.717930 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:26.718037 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:26.718106 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:26.718127 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 14:56:26.718144 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:56:26.720206 4600 raft_consensus.cc:1261] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Refusing update from remote peer 5b5c325117754e3482bd9734bf13076b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:26.720232 4332 raft_consensus.cc:1261] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Refusing update from remote peer 5b5c325117754e3482bd9734bf13076b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:26.720585 4669 consensus_queue.cc:1035] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:26.720774 4673 consensus_queue.cc:1035] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:26.724282 4677 mvcc.cc:205] Tried to move back new op lower bound from 7333266172802617344 to 7333266172720115712. Current Snapshot: MvccSnapshot[applied={T|T < 7333266172802617344}]
I20260925 14:56:26.724592 4678 mvcc.cc:205] Tried to move back new op lower bound from 7333266172802617344 to 7333266172720115712. Current Snapshot: MvccSnapshot[applied={T|T < 7333266172802617344 or (T in {7333266172802617344})}]
I20260925 14:56:26.724696 4676 mvcc.cc:205] Tried to move back new op lower bound from 7333266172802617344 to 7333266172720115712. Current Snapshot: MvccSnapshot[applied={T|T < 7333266172802617344 or (T in {7333266172802617344})}]
I20260925 14:56:26.725637 4187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:26.725734 4187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:26.726534 4187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Sending DeleteTablet for 3 replicas of tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca
I20260925 14:56:26.726903 4446 tablet_service.cc:1514] Processing DeleteTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:26 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:51870
I20260925 14:56:26.726920 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:26 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:56298
I20260925 14:56:26.727144 4687 tablet_replica.cc:330] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: stopping tablet replica
I20260925 14:56:26.727239 4687 raft_consensus.cc:2229] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:26.727233 4312 tablet_service.cc:1514] Processing DeleteTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:26 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:58578
I20260925 14:56:26.727284 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4249
I20260925 14:56:26.727339 4687 raft_consensus.cc:2258] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:26.728672 4687 ts_tablet_manager.cc:1905] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:26.730365 4687 ts_tablet_manager.cc:1918] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:26.730453 4687 log.cc:1196] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/e50c9b4d45b14fcb81e4c8dd25a7a0ca
I20260925 14:56:26.730705 4687 ts_tablet_manager.cc:1939] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Deleting consensus metadata
I20260925 14:56:26.731128 4173 catalog_manager.cc:4841] TS 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191): tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca (table test-table [id=dd3721c67a9a42eab39ac52ca6a1fb1f]) successfully deleted
I20260925 14:56:26.733557 4686 tablet_replica.cc:330] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: stopping tablet replica
I20260925 14:56:26.733683 4686 raft_consensus.cc:2229] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:26.733793 4686 raft_consensus.cc:2258] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:26.734256 4686 ts_tablet_manager.cc:1905] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:56:26.734436 4171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:34943 (error 108)
I20260925 14:56:26.734961 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.734814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.0.0.1:58586)
0925 14:56:26.734874 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.734921 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:56:26.735136 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4383
W20260925 14:56:26.735183 4171 catalog_manager.cc:4584] TS dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34943: connect: Connection refused (error 111)
I20260925 14:56:26.740413 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4517
I20260925 14:56:26.740444 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:26.740298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:38635 (local address 127.0.0.1:51884)
0925 14:56:26.740370 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:26.740403 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.2:38635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:56:26.745455 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4155
I20260925 14:56:46.750913 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45219
--webserver_interface=127.20.199.62
--webserver_port=43871
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:46.832288 4689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:46.832521 4689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:46.832558 4689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:46.833720 4689 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:56:46.833783 4689 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:56:46.833806 4689 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:56:46.833825 4689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:46.833887 4689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:46.833918 4689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:46.833945 4689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:46.833971 4689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:46.835421 4689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:46.835701 4689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:46.835963 4689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:56:46.844518 4689 server_base.cc:1034] running on GCE node
W20260925 14:56:46.844585 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:46.844638 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:46.844588 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:46.845080 4689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:46.845350 4689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:46.846536 4689 hybrid_clock.cc:648] HybridClock initialized: now 1790348206846521 us; error 33 us; skew 500 ppm
Sep 25 14:56:46 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348206, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:46.858618 4689 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:56:46.859112 4689 webserver.cc:460] Webserver started at http://127.20.199.62:43871/ using document root <none> and password file <none>
I20260925 14:56:46.859318 4689 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:46.859375 4689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:46.860801 4689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:46.861505 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:46.861694 4689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:46.861820 4689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "b55b65bb22ce4b67bc75bb410820bb0e"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "3a48f6658fc5a18f3bbeddd930d3d513"
server_key_iv: "430f9097b3d4e5968fc905aff86be80c"
server_key_version: "encryptionkey@0"
I20260925 14:56:46.862228 4689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:46.894757 4689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:46.897142 4689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:46.897261 4689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:46.901513 4689 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:45219
I20260925 14:56:46.901566 4757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:45219 every 8 connection(s)
I20260925 14:56:46.901964 4689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:46.903548 4758 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:56:46.904207 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap starting.
I20260925 14:56:46.906599 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4689
I20260925 14:56:46.906945 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:34943
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:45219
--webserver_port=43771
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:46.913473 4758 log.cc:824] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:46.915495 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:46.915851 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap complete.
I20260925 14:56:46.918296 4758 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:46.918530 4758 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55b65bb22ce4b67bc75bb410820bb0e, State: Initialized, Role: FOLLOWER
I20260925 14:56:46.918696 4758 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:46.918799 4758 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:46.918885 4758 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:46.918954 4758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:56:46.919997 4758 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:46.920125 4758 leader_election.cc:302] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55b65bb22ce4b67bc75bb410820bb0e; no voters:
I20260925 14:56:46.920353 4758 leader_election.cc:288] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:56:46.920390 4762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:56:46.920507 4762 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 LEADER]: Becoming Leader. State: Replica: b55b65bb22ce4b67bc75bb410820bb0e, State: Running, Role: LEADER
I20260925 14:56:46.920595 4762 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:56:46.920631 4758 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:46.920858 4762 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55b65bb22ce4b67bc75bb410820bb0e. Latest consensus state: current_term: 2 leader_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } } }
I20260925 14:56:46.920926 4762 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:46.921459 4767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:46.920892 4763 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } } }
I20260925 14:56:46.921635 4763 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:46.922341 4767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:46.922464 4767 catalog_manager.cc:1259] Loaded cluster ID: bab5139583264ea3929ed3756b28af8d
I20260925 14:56:46.922735 4767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:46.923547 4767 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:46.923758 4767 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Loaded TSK: 0
I20260925 14:56:46.929490 4767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Generated new TSK 1
I20260925 14:56:46.929570 4767 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:56:47.011685 4760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:47.011929 4760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:47.011960 4760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:47.013325 4760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:47.013378 4760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:47.013473 4760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:56:47.015173 4760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=43771
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:47.015486 4760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:47.015743 4760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:47.020991 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:47.021162 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.021014 4760 server_base.cc:1034] running on GCE node
W20260925 14:56:47.021039 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.021570 4760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:47.021888 4760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:47.023073 4760 hybrid_clock.cc:648] HybridClock initialized: now 1790348207023044 us; error 46 us; skew 500 ppm
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:47.026315 4760 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:56:47.026856 4760 webserver.cc:460] Webserver started at http://127.20.199.1:43771/ using document root <none> and password file <none>
I20260925 14:56:47.027107 4760 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:47.027168 4760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:47.028788 4760 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:56:47.029711 4795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.029933 4760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:47.030035 4760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "e0f3b66185a800fa78bd116aea32f237"
server_key_iv: "76d8267c50d6657fca0ee733d5b83e61"
server_key_version: "encryptionkey@0"
I20260925 14:56:47.030503 4760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:47.048380 4760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:47.051119 4760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:47.051287 4760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:47.051555 4760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:47.052094 4802 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:56:47.053884 4760 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:56:47.053947 4760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:56:47.053987 4760 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:56:47.054904 4760 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:56:47.054955 4760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:47.055037 4802 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap starting.
I20260925 14:56:47.062644 4760 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:34943
I20260925 14:56:47.062819 4909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:34943 every 8 connection(s)
I20260925 14:56:47.063119 4760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:47.063550 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4760
I20260925 14:56:47.063930 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:38635
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:45219
--webserver_port=42935
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:47.066629 4802 log.cc:824] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:47.069546 4802 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:56:47.069911 4802 tablet_bootstrap.cc:492] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap complete.
I20260925 14:56:47.071261 4802 ts_tablet_manager.cc:1397] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260925 14:56:47.072731 4802 raft_consensus.cc:348] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.073087 4802 raft_consensus.cc:729] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba34c3b84cc431aa6a5fa4d6428e848, State: Initialized, Role: FOLLOWER
I20260925 14:56:47.073223 4802 consensus_queue.cc:260] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.073516 4802 ts_tablet_manager.cc:1428] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:47.085487 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.066352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38489 (local address 127.20.199.62:45219)
0925 14:56:47.069683 (+ 3331us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.069698 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.071216 (+ 1518us) server_negotiation.cc:407] Connection header received
0925 14:56:47.071464 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.071473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.071545 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.071702 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.072812 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.074956 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.076533 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.077243 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.077372 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.080312 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.080317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.080318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.080333 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.083284 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.083836 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.083839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.083840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.083991 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.084310 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.084311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.084312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.084634 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.084731 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.085104 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.085250 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3184,"thread_start_us":77,"threads_started":1}
I20260925 14:56:47.085498 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.066306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:38489)
0925 14:56:47.070115 (+ 3809us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.070168 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.071022 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.071865 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.071870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.072331 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:47.072653 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.072661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.075097 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.075098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.076420 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.076423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.077366 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.077367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.077539 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.078209 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:47.078227 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:47.080168 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:47.083392 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.083399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.083401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.083723 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.084085 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.084086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.084086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.084226 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.084720 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:47.084725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:47.084904 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.085024 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.085199 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3756,"thread_start_us":100,"threads_started":1}
I20260925 14:56:47.086715 4910 heartbeater.cc:347] Connected to a master server at 127.20.199.62:45219
I20260925 14:56:47.086823 4910 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:47.087004 4910 heartbeater.cc:511] Master 127.20.199.62:45219 requested a full tablet report, sending...
I20260925 14:56:47.087816 4722 ts_manager.cc:194] Registered new tserver with Master: dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:56:47.089071 4722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:38489
I20260925 14:56:47.090138 4910 heartbeater.cc:503] Master 127.20.199.62:45219 was elected leader, sending a full tablet report...
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348206, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:56:47.099316 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.088780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.20.199.1:34943)
0925 14:56:47.089295 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.089300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.089420 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:56:47.089480 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.089483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.089500 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.090384 (+ 884us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.090999 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.092148 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.093429 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.093885 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.093966 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.096278 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.096285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.096285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.096292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.098051 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.098458 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.098461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.098461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.098521 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.098731 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.098732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.098732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.098863 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.098946 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.099128 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.099210 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":88,"threads_started":1}
I20260925 14:56:47.099323 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.088452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.0.0.1:35772)
0925 14:56:47.088960 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.089000 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.089066 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.090442 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.090449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.090845 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:47.090903 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.090906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.092241 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.092242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.093321 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.093324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.093975 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.093976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.094054 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.094619 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:47.094625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:47.096168 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:47.098158 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.098165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.098167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.098370 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.098596 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.098597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.098597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.098661 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.098941 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:47.098947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:47.098996 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.099093 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.099210 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":98,"threads_started":1}
I20260925 14:56:47.100057 4844 tablet_service.cc:1514] Processing DeleteTablet for tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:26 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:35772
I20260925 14:56:47.100776 4919 tablet_replica.cc:330] stopping tablet replica
I20260925 14:56:47.100910 4919 raft_consensus.cc:2229] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:47.100999 4919 raft_consensus.cc:2258] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:47.101295 4919 ts_tablet_manager.cc:1905] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:47.103541 4919 ts_tablet_manager.cc:1918] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:47.103660 4919 log.cc:1196] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal/wals/e50c9b4d45b14fcb81e4c8dd25a7a0ca
I20260925 14:56:47.103940 4919 ts_tablet_manager.cc:1939] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P dba34c3b84cc431aa6a5fa4d6428e848: Deleting consensus metadata
I20260925 14:56:47.104485 4706 catalog_manager.cc:4841] TS dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943): tablet e50c9b4d45b14fcb81e4c8dd25a7a0ca (table test-table [id=dd3721c67a9a42eab39ac52ca6a1fb1f]) successfully deleted
W20260925 14:56:47.162914 4912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:47.163125 4912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:47.163149 4912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:47.164409 4912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:47.164455 4912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:47.164484 4912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:56:47.166224 4912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:38635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=42935
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:47.166476 4912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:47.166744 4912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:47.172122 4924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:47.172133 4925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.172333 4912 server_base.cc:1034] running on GCE node
W20260925 14:56:47.172135 4927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.172626 4912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:47.172930 4912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:47.174077 4912 hybrid_clock.cc:648] HybridClock initialized: now 1790348207174055 us; error 37 us; skew 500 ppm
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:47.176388 4912 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:56:47.176762 4912 webserver.cc:460] Webserver started at http://127.20.199.2:42935/ using document root <none> and password file <none>
I20260925 14:56:47.176941 4912 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:47.176991 4912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:47.178303 4912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:47.178917 4934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.179103 4912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:56:47.179198 4912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "5b5c325117754e3482bd9734bf13076b"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "9d0e533bcc62fcb6775fba0931d5b719"
server_key_iv: "11324f682472247f7b9756212bfd9e3c"
server_key_version: "encryptionkey@0"
I20260925 14:56:47.179530 4912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:47.190864 4912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:47.192669 4912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:47.192821 4912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:47.193048 4912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:47.193560 4941 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 14:56:47.194932 4941 ts_tablet_manager.cc:1852] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 14:56:47.195214 4941 ts_tablet_manager.cc:1905] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:47.197456 4941 ts_tablet_manager.cc:1918] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:47.197576 4941 log.cc:1196] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/e50c9b4d45b14fcb81e4c8dd25a7a0ca
I20260925 14:56:47.197889 4941 ts_tablet_manager.cc:1939] T e50c9b4d45b14fcb81e4c8dd25a7a0ca P 5b5c325117754e3482bd9734bf13076b: Deleting consensus metadata
I20260925 14:56:47.198100 4912 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 14:56:47.198143 4912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 14:56:47.198168 4912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:47.198181 4912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.204126 4912 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:38635
I20260925 14:56:47.204226 5048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:38635 every 8 connection(s)
I20260925 14:56:47.204519 4912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:47.210628 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 4912
I20260925 14:56:47.210928 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:33191
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:45219
--webserver_port=42305
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:56:47.214272 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.205272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:47839 (local address 127.20.199.62:45219)
0925 14:56:47.205348 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.205350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.205588 (+ 238us) server_negotiation.cc:407] Connection header received
0925 14:56:47.206150 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.206152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.206164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.206220 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.206845 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.207367 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.208372 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.208586 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.208673 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.211196 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.211197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.211198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.211206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.212681 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.213134 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.213135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.213136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.213195 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.213502 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.213503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.213504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.213622 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.213701 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.214088 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.214181 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:56:47.214473 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.205248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:47839)
0925 14:56:47.205466 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.205501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.206039 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.206299 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.206303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.206546 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:47.206731 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.206734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.207460 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.207461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.208267 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.208268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.208681 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.208682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.208815 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.209371 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:47.209379 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:47.211089 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:47.212812 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.212818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.212820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.213040 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.213280 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.213281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.213281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.213396 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.213692 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:47.213698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:47.213875 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.214050 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.214221 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260925 14:56:47.214984 5049 heartbeater.cc:347] Connected to a master server at 127.20.199.62:45219
I20260925 14:56:47.215066 5049 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:47.215260 5049 heartbeater.cc:511] Master 127.20.199.62:45219 requested a full tablet report, sending...
I20260925 14:56:47.215687 4722 ts_manager.cc:194] Registered new tserver with Master: 5b5c325117754e3482bd9734bf13076b (127.20.199.2:38635)
I20260925 14:56:47.216080 4722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:47839
W20260925 14:56:47.294375 5052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:47.294593 5052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:47.294631 5052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:47.295769 5052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:47.295835 5052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:47.295879 5052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:56:47.297314 5052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:33191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=42305
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:56:47.297585 5052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:47.297807 5052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:47.302591 5058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:47.302585 5060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.302775 5052 server_base.cc:1034] running on GCE node
W20260925 14:56:47.302814 5057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:47.303134 5052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:47.303390 5052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:47.304565 5052 hybrid_clock.cc:648] HybridClock initialized: now 1790348207304563 us; error 40 us; skew 500 ppm
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:47.307091 5052 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:56:47.307554 5052 webserver.cc:460] Webserver started at http://127.20.199.3:42305/ using document root <none> and password file <none>
I20260925 14:56:47.307792 5052 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:47.307883 5052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:47.309197 5052 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:47.309878 5067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.310046 5052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.310137 5052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "2b5dd4eab3171166941c5ca52c675e93"
server_key_iv: "4b339df5875e3a750c9eedbb8350549c"
server_key_version: "encryptionkey@0"
I20260925 14:56:47.310487 5052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:47.331110 5052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:47.333361 5052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:47.333590 5052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:47.333891 5052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:47.334252 5052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:47.334317 5052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.334365 5052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:47.334425 5052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:47.341645 5052 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:33191
I20260925 14:56:47.341686 5180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:33191 every 8 connection(s)
I20260925 14:56:47.342108 5052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:47.346081 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5052
Sep 25 14:56:47 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:56:47.352468 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.342880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:51719 (local address 127.20.199.62:45219)
0925 14:56:47.342961 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.342964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.343265 (+ 301us) server_negotiation.cc:407] Connection header received
0925 14:56:47.343720 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.343721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.343737 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.343811 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.344741 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.345290 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.346552 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.346869 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.346933 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.349212 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.349213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.349213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.349221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.350945 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.351450 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.351451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.351451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.351506 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.351811 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.351813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.351813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.351928 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.352008 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.352300 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.352380 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:56:47.352631 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.342889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:51719)
0925 14:56:47.343126 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.343162 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.343625 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.343881 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.343885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.344458 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:47.344649 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.344654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.345438 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.345438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.346443 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.346446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.346951 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.346952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.347092 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.347639 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:47.347647 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:47.349116 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:47.351020 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.351027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.351029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.351370 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.351592 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:47.351593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:47.351593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:47.351707 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:47.351999 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:47.352006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:47.352145 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.352247 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.352402 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
I20260925 14:56:47.353048 5181 heartbeater.cc:347] Connected to a master server at 127.20.199.62:45219
I20260925 14:56:47.353142 5181 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:47.353344 5181 heartbeater.cc:511] Master 127.20.199.62:45219 requested a full tablet report, sending...
I20260925 14:56:47.353433 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.346512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.20.199.62:45219)
0925 14:56:47.346641 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.346643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.346815 (+ 172us) server_negotiation.cc:407] Connection header received
0925 14:56:47.346876 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.346877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.346886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.346969 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.347532 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.348008 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.348844 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.349136 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.349199 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.350724 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.350725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.350726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.350731 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.352264 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.352664 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.352665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.352666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.352725 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.352921 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.352922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.352922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.353050 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.353212 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.353251 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.353327 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260925 14:56:47.353739 4721 ts_manager.cc:194] Registered new tserver with Master: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191)
I20260925 14:56:47.354238 4721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:51719
I20260925 14:56:47.354285 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:47.356925 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.354620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.20.199.62:45219)
0925 14:56:47.354654 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.354656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.354727 (+ 71us) server_negotiation.cc:407] Connection header received
0925 14:56:47.354790 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.354790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.354876 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.354953 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:47.355096 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.355590 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.356351 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.356544 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.356605 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.356734 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:47.356769 (+ 35us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 14:56:47.356886 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37024: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:56:47.357002 5184 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37024: token expired
W20260925 14:56:47.357115 4650 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.199.62:45219)
I20260925 14:56:47.362586 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.357383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.20.199.62:45219)
0925 14:56:47.357451 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.357453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.357460 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:56:47.357477 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.357478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.357486 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.357536 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:47.357948 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.358370 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.358996 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.359157 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.359207 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.360197 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:47.360198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:47.360198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:47.360204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:47.361381 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.361790 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.361791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.361791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.361885 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:47.362093 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:47.362094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:47.362094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:47.362200 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:47.362299 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.362427 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.362495 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:56:47.363579 4721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:47.365315 4721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:47.367446 4844 tablet_service.cc:1467] Processing CreateTablet for tablet 37153332b42346bab4505195a35c7af8 (DEFAULT_TABLE table=test-table [id=24a048b82b0b42cb857132209bc2ed61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:47.367642 4844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37153332b42346bab4505195a35c7af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:47.369376 4802 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap starting.
I20260925 14:56:47.369886 4802 tablet_bootstrap.cc:654] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:47.370525 4802 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: No bootstrap required, opened a new log
I20260925 14:56:47.370590 4802 ts_tablet_manager.cc:1397] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:47.370941 4802 raft_consensus.cc:348] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.371018 4802 raft_consensus.cc:374] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:47.371061 4802 raft_consensus.cc:729] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba34c3b84cc431aa6a5fa4d6428e848, State: Initialized, Role: FOLLOWER
I20260925 14:56:47.371112 4802 consensus_queue.cc:260] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.371199 4802 ts_tablet_manager.cc:1428] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:47.371833 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.367304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:38635 (local address 127.0.0.1:51894)
0925 14:56:47.367435 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.367444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.367481 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.367881 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.367882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.367884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:47.367930 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.367931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.369035 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.369037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.370563 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.370565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.371568 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.371569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.371645 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.371660 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.371711 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.371754 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260925 14:56:47.371847 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.367481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.20.199.2:38635)
0925 14:56:47.367653 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.367658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.367668 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:56:47.367707 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.367710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.367725 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.367809 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:47.368042 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.368947 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.370656 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.371392 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.371445 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.371506 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.371728 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.371769 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260925 14:56:47.372510 4983 tablet_service.cc:1467] Processing CreateTablet for tablet 37153332b42346bab4505195a35c7af8 (DEFAULT_TABLE table=test-table [id=24a048b82b0b42cb857132209bc2ed61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:47.372861 4983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37153332b42346bab4505195a35c7af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:47.374307 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.367175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33191 (local address 127.0.0.1:45574)
0925 14:56:47.367308 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.367319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.367360 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.369637 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.369638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.369641 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:47.369696 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.369698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.370902 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.370903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.371899 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.371901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.374105 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.374106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.374161 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.374164 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.374217 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.374252 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 14:56:47.374364 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.367254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.20.199.3:33191)
0925 14:56:47.369449 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.369453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.369479 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:56:47.369519 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.369521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.369537 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.369623 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:47.369760 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.370816 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.371980 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.374014 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.374096 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.374148 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.374230 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.374279 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2132,"thread_start_us":107,"threads_started":1}
I20260925 14:56:47.374948 4941 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Bootstrap starting.
I20260925 14:56:47.375209 5115 tablet_service.cc:1467] Processing CreateTablet for tablet 37153332b42346bab4505195a35c7af8 (DEFAULT_TABLE table=test-table [id=24a048b82b0b42cb857132209bc2ed61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:47.375525 5115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37153332b42346bab4505195a35c7af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:47.375828 4941 tablet_bootstrap.cc:654] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:47.376119 4941 log.cc:824] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:47.376766 4941 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: No bootstrap required, opened a new log
I20260925 14:56:47.376858 4941 ts_tablet_manager.cc:1397] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:47.378153 5192 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Bootstrap starting.
I20260925 14:56:47.378485 4941 raft_consensus.cc:348] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.378662 4941 raft_consensus.cc:374] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:47.378697 4941 raft_consensus.cc:729] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5c325117754e3482bd9734bf13076b, State: Initialized, Role: FOLLOWER
I20260925 14:56:47.378844 5192 tablet_bootstrap.cc:654] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:47.378837 4941 consensus_queue.cc:260] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.379132 4941 ts_tablet_manager.cc:1428] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:56:47.379302 5049 heartbeater.cc:503] Master 127.20.199.62:45219 was elected leader, sending a full tablet report...
I20260925 14:56:47.379424 5192 log.cc:824] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:47.380270 5192 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: No bootstrap required, opened a new log
I20260925 14:56:47.380349 5192 ts_tablet_manager.cc:1397] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:47.381831 5192 raft_consensus.cc:348] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.381973 5192 raft_consensus.cc:374] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:47.382032 5192 raft_consensus.cc:729] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6, State: Initialized, Role: FOLLOWER
I20260925 14:56:47.382154 5192 consensus_queue.cc:260] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.382447 5192 ts_tablet_manager.cc:1428] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:47.382452 5181 heartbeater.cc:503] Master 127.20.199.62:45219 was elected leader, sending a full tablet report...
I20260925 14:56:47.389302 5193 raft_consensus.cc:482] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:47.389384 5193 raft_consensus.cc:504] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.389739 5193 leader_election.cc:288] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191), dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:56:47.392521 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.390081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:43509 (local address 127.20.199.3:33191)
0925 14:56:47.390135 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.390137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.390145 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:47.390209 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.390210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.390220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.390281 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:47.390444 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.390878 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.391804 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.392152 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.392206 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.392213 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.392408 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.392445 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:56:47.392529 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.389920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33191 (local address 127.20.199.2:43509)
0925 14:56:47.390090 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.390100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.390148 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.390309 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.390311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.390312 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:47.390379 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.390380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.390978 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.390978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.391695 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.391696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.392232 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.392234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.392310 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.392315 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.392398 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.392440 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260925 14:56:47.392619 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.389952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:42315 (local address 127.20.199.1:34943)
0925 14:56:47.390023 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.390025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.390033 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:47.390066 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.390067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.390094 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.390168 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:47.390328 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.390873 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.391770 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.392241 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.392340 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.392418 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.392506 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.392545 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:56:47.392817 5135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37153332b42346bab4505195a35c7af8" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" is_pre_election: true
I20260925 14:56:47.392926 5135 raft_consensus.cc:2454] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 0.
I20260925 14:56:47.393085 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.389885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.20.199.2:42315)
0925 14:56:47.389957 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.389969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:47.390008 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:47.390173 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:47.390174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:47.390175 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:47.390248 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.390251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.390956 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.390957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.391681 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:47.391683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.392356 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.392357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:47.392434 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.392437 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:56:47.392499 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:56:47.392539 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:56:47.393110 4937 leader_election.cc:302] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6, 5b5c325117754e3482bd9734bf13076b; no voters:
I20260925 14:56:47.393241 5193 raft_consensus.cc:2781] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:47.393289 5193 raft_consensus.cc:482] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:47.393312 5193 raft_consensus.cc:3037] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:47.393430 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37153332b42346bab4505195a35c7af8" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" is_pre_election: true
I20260925 14:56:47.393584 4864 raft_consensus.cc:2454] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 0.
I20260925 14:56:47.394095 5193 raft_consensus.cc:504] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.394230 5193 leader_election.cc:288] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 election: Requested vote from peers 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191), dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:56:47.394363 5135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37153332b42346bab4505195a35c7af8" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
I20260925 14:56:47.394390 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37153332b42346bab4505195a35c7af8" candidate_uuid: "5b5c325117754e3482bd9734bf13076b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
I20260925 14:56:47.394439 5135 raft_consensus.cc:3037] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:47.394452 4864 raft_consensus.cc:3037] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:47.395248 5135 raft_consensus.cc:2454] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 1.
I20260925 14:56:47.395248 4864 raft_consensus.cc:2454] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5c325117754e3482bd9734bf13076b in term 1.
I20260925 14:56:47.395421 4937 leader_election.cc:302] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6, 5b5c325117754e3482bd9734bf13076b; no voters:
I20260925 14:56:47.395519 5193 raft_consensus.cc:2781] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:47.395654 5193 raft_consensus.cc:686] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 LEADER]: Becoming Leader. State: Replica: 5b5c325117754e3482bd9734bf13076b, State: Running, Role: LEADER
I20260925 14:56:47.395740 5193 consensus_queue.cc:237] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:56:47.396471 4721 catalog_manager.cc:5495] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5c325117754e3482bd9734bf13076b (127.20.199.2). New cstate: current_term: 1 leader_uuid: "5b5c325117754e3482bd9734bf13076b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:47.406178 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.404123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.20.199.2:38635)
0925 14:56:47.404184 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:47.404185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:47.404193 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:47.404218 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:47.404219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:47.404234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:47.404291 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:47.404404 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.404842 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.405573 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:47.405754 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:47.405806 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:47.405930 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:47.406013 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:47.406065 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:47.406100 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:56:47.406115 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:56:47.408085 4864 raft_consensus.cc:1261] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Refusing update from remote peer 5b5c325117754e3482bd9734bf13076b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:47.408133 5135 raft_consensus.cc:1261] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Refusing update from remote peer 5b5c325117754e3482bd9734bf13076b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:47.408432 5193 consensus_queue.cc:1035] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:47.408576 5197 consensus_queue.cc:1035] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:47.411347 5202 mvcc.cc:205] Tried to move back new op lower bound from 7333266257540648960 to 7333266257493716992. Current Snapshot: MvccSnapshot[applied={T|T < 7333266257540648960}]
I20260925 14:56:47.413061 4721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:47.413157 4721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:47.413983 4721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Sending DeleteTablet for 3 replicas of tablet 37153332b42346bab4505195a35c7af8
I20260925 14:56:47.414319 4844 tablet_service.cc:1514] Processing DeleteTablet for tablet 37153332b42346bab4505195a35c7af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:47 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:35772
I20260925 14:56:47.414336 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4760
I20260925 14:56:47.414409 4919 tablet_replica.cc:330] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: stopping tablet replica
I20260925 14:56:47.414381 5115 tablet_service.cc:1514] Processing DeleteTablet for tablet 37153332b42346bab4505195a35c7af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:47 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:45574
I20260925 14:56:47.414453 4919 raft_consensus.cc:2229] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:47.414505 4919 raft_consensus.cc:2258] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:47.414903 4983 tablet_service.cc:1514] Processing DeleteTablet for tablet 37153332b42346bab4505195a35c7af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:47 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:51894
I20260925 14:56:47.415199 5208 tablet_replica.cc:330] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: stopping tablet replica
I20260925 14:56:47.415313 5208 raft_consensus.cc:2229] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:47.415418 5208 raft_consensus.cc:2258] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:47.416478 5208 ts_tablet_manager.cc:1905] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:47.418323 5208 ts_tablet_manager.cc:1918] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:47.418398 5208 log.cc:1196] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/37153332b42346bab4505195a35c7af8
I20260925 14:56:47.418680 5208 ts_tablet_manager.cc:1939] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Deleting consensus metadata
I20260925 14:56:47.419147 4708 catalog_manager.cc:4841] TS 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191): tablet 37153332b42346bab4505195a35c7af8 (table test-table [id=24a048b82b0b42cb857132209bc2ed61]) successfully deleted
W20260925 14:56:47.420650 4706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:34943 (error 108)
I20260925 14:56:47.420933 5209 tablet_replica.cc:330] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: stopping tablet replica
I20260925 14:56:47.421051 5209 raft_consensus.cc:2229] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:47.421111 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.420957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.0.0.1:35782)
0925 14:56:47.421037 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.421080 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 14:56:47.421154 5209 raft_consensus.cc:2258] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:47.421344 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4912
W20260925 14:56:47.421213 4706 catalog_manager.cc:4584] TS dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37153332b42346bab4505195a35c7af8: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34943: connect: Connection refused (error 111)
I20260925 14:56:47.426041 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 5052
I20260925 14:56:47.426093 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:47.425967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:38635 (local address 127.0.0.1:51914)
0925 14:56:47.426026 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:47.426059 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.2:38635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:56:47.430653 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 4689
I20260925 14:57:07.436425 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45219
--webserver_interface=127.20.199.62
--webserver_port=43871
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:07.518554 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:07.518800 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:07.518839 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:07.519994 5211 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:57:07.520053 5211 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:57:07.520074 5211 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:57:07.520094 5211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:07.520135 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:07.520167 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:07.520191 5211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:07.520217 5211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:07.521771 5211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:07.522044 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:07.522260 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:07.527208 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.527196 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.527261 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:07.527379 5211 server_base.cc:1034] running on GCE node
I20260925 14:57:07.527645 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:07.528033 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:07.529189 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790348227529182 us; error 34 us; skew 500 ppm
Sep 25 14:57:07 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:07.536253 5211 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:07.536731 5211 webserver.cc:460] Webserver started at http://127.20.199.62:43871/ using document root <none> and password file <none>
I20260925 14:57:07.536976 5211 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:07.537053 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:07.538484 5211 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:07.539187 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:07.539378 5211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:07.539484 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "b55b65bb22ce4b67bc75bb410820bb0e"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "3a48f6658fc5a18f3bbeddd930d3d513"
server_key_iv: "430f9097b3d4e5968fc905aff86be80c"
server_key_version: "encryptionkey@0"
I20260925 14:57:07.539834 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:07.565004 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:07.567029 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:07.567194 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:07.571367 5211 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:45219
I20260925 14:57:07.571399 5278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:45219 every 8 connection(s)
I20260925 14:57:07.571847 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:07.573371 5279 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:57:07.574010 5279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap starting.
I20260925 14:57:07.582005 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5211
I20260925 14:57:07.582376 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:34943
--local_ip_for_outbound_sockets=127.20.199.1
--tserver_master_addrs=127.20.199.62:45219
--webserver_port=43771
--webserver_interface=127.20.199.1
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:07.582748 5279 log.cc:824] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:07.585737 5279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:07.586089 5279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Bootstrap complete.
I20260925 14:57:07.590320 5279 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:57:07.590724 5279 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55b65bb22ce4b67bc75bb410820bb0e, State: Initialized, Role: FOLLOWER
I20260925 14:57:07.590940 5279 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:57:07.591037 5279 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:07.591101 5279 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:07.591166 5279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:57:07.592890 5279 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:57:07.593022 5279 leader_election.cc:302] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55b65bb22ce4b67bc75bb410820bb0e; no voters:
I20260925 14:57:07.593215 5279 leader_election.cc:288] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:57:07.593299 5283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:57:07.593549 5279 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:07.593549 5283 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [term 3 LEADER]: Becoming Leader. State: Replica: b55b65bb22ce4b67bc75bb410820bb0e, State: Running, Role: LEADER
I20260925 14:57:07.593792 5283 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } }
I20260925 14:57:07.594074 5284 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } } }
I20260925 14:57:07.594148 5284 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:07.594313 5285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55b65bb22ce4b67bc75bb410820bb0e. Latest consensus state: current_term: 3 leader_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55b65bb22ce4b67bc75bb410820bb0e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45219 } } }
I20260925 14:57:07.594374 5289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:07.594440 5285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:07.595280 5289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:07.595391 5289 catalog_manager.cc:1259] Loaded cluster ID: bab5139583264ea3929ed3756b28af8d
I20260925 14:57:07.595474 5289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:07.596580 5289 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:07.596761 5289 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Loaded TSK: 0
I20260925 14:57:07.596817 5289 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b55b65bb22ce4b67bc75bb410820bb0e: Loaded TSK: 1
F20260925 14:57:07.597039 5289 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ee11e09ad google::LogMessage::Fail() at ??:0
@ 0x7f1ee11e388a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ee11e04de google::LogMessage::Flush() at ??:0
@ 0x7f1ee11e1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1ee394d104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1ee394ad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1ee913a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1ee913a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1ee913a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1ee9100c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1ee910aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1ee30ae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1ee30a3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1ee3f076db start_thread at ??:0
@ 0x7f1edfcde61f clone at ??:0
W20260925 14:57:07.682943 5282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:07.683175 5282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:07.683216 5282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:07.684371 5282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:07.684443 5282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:07.684492 5282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:07.686012 5282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=43771
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:07.686295 5282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:07.686530 5282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:07.692476 5321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.692443 5320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.692677 5323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:07.692764 5282 server_base.cc:1034] running on GCE node
I20260925 14:57:07.692945 5282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:07.693214 5282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:07.694398 5282 hybrid_clock.cc:648] HybridClock initialized: now 1790348227694365 us; error 46 us; skew 500 ppm
Sep 25 14:57:07 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:07.697326 5282 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:57:07.697796 5282 webserver.cc:460] Webserver started at http://127.20.199.1:43771/ using document root <none> and password file <none>
I20260925 14:57:07.698051 5282 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:07.698122 5282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:07.699586 5282 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:07.700347 5330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:07.700523 5282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:07.700628 5282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "dba34c3b84cc431aa6a5fa4d6428e848"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "e0f3b66185a800fa78bd116aea32f237"
server_key_iv: "76d8267c50d6657fca0ee733d5b83e61"
server_key_version: "encryptionkey@0"
I20260925 14:57:07.700989 5282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:07.722810 5282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:07.725215 5282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:07.725383 5282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:07.725690 5282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:07.726145 5337 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:57:07.728165 5282 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:57:07.728232 5282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:57:07.728276 5282 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:57:07.729041 5282 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:57:07.729094 5282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:07.729172 5337 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap starting.
I20260925 14:57:07.736438 5282 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:34943
I20260925 14:57:07.736578 5444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:34943 every 8 connection(s)
I20260925 14:57:07.736861 5282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:07.739099 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5282
I20260925 14:57:07.739399 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:38635
--local_ip_for_outbound_sockets=127.20.199.2
--tserver_master_addrs=127.20.199.62:45219
--webserver_port=42935
--webserver_interface=127.20.199.2
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:07.740937 5337 log.cc:824] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:07.743901 5337 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:07.744246 5337 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Bootstrap complete.
I20260925 14:57:07.745633 5337 ts_tablet_manager.cc:1397] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20260925 14:57:07.747274 5337 raft_consensus.cc:348] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:07.747642 5337 raft_consensus.cc:729] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba34c3b84cc431aa6a5fa4d6428e848, State: Initialized, Role: FOLLOWER
I20260925 14:57:07.747851 5337 consensus_queue.cc:260] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:07.748169 5337 ts_tablet_manager.cc:1428] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260925 14:57:07.800679 5447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:07.737745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:47407)
0925 14:57:07.737988 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.738024 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:07.738468 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:07.800475 (+ 62007us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":132,"threads_started":1}
W20260925 14:57:07.800891 5445 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:45219 (0 consecutive failures): Network error: Failed to ping master at 127.20.199.62:45219: Client connection negotiation failed: client connection to 127.20.199.62:45219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:57:07.801271 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.801144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:43819)
0925 14:57:07.801206 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.801251 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:57:07.801618 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.801523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:58067)
0925 14:57:07.801571 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.801594 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:57:07.801728 5445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:57:07.825317 5448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:07.825599 5448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:07.825625 5448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:07.826797 5448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:07.826838 5448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:07.826864 5448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:07.828327 5448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:38635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=42935
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:07.828639 5448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:07.828868 5448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:07.833896 5456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.833897 5458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.833932 5455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:07.833928 5448 server_base.cc:1034] running on GCE node
I20260925 14:57:07.834316 5448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:07.834558 5448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:07.835717 5448 hybrid_clock.cc:648] HybridClock initialized: now 1790348227835715 us; error 36 us; skew 500 ppm
Sep 25 14:57:07 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:07.838172 5448 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:57:07.838593 5448 webserver.cc:460] Webserver started at http://127.20.199.2:42935/ using document root <none> and password file <none>
I20260925 14:57:07.838814 5448 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:07.838886 5448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:07.840193 5448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:07.840867 5465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:07.841054 5448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:07.841166 5448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "5b5c325117754e3482bd9734bf13076b"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "9d0e533bcc62fcb6775fba0931d5b719"
server_key_iv: "11324f682472247f7b9756212bfd9e3c"
server_key_version: "encryptionkey@0"
I20260925 14:57:07.841573 5448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:07.860785 5448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:07.862767 5448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:07.862946 5448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:07.863191 5448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:07.863673 5472 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:57:07.865181 5448 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:57:07.865243 5448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:57:07.865283 5448 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:57:07.865954 5448 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:57:07.866011 5448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:07.866140 5472 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Bootstrap starting.
I20260925 14:57:07.872598 5448 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:38635
I20260925 14:57:07.872694 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:38635 every 8 connection(s)
I20260925 14:57:07.873015 5448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:07.874051 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.873659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:38707)
0925 14:57:07.873860 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.873930 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
W20260925 14:57:07.874284 5580 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:45219 (0 consecutive failures): Network error: Failed to ping master at 127.20.199.62:45219: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
I20260925 14:57:07.874554 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.874461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:33297)
0925 14:57:07.874508 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.874532 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:57:07.874905 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.874817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:47961)
0925 14:57:07.874864 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.874886 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:57:07.875046 5580 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:57:07.877111 5472 log.cc:824] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:07.877239 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5448
I20260925 14:57:07.877583 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:33191
--local_ip_for_outbound_sockets=127.20.199.3
--tserver_master_addrs=127.20.199.62:45219
--webserver_port=42305
--webserver_interface=127.20.199.3
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:07.879487 5472 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:07.879786 5472 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Bootstrap complete.
I20260925 14:57:07.880726 5472 ts_tablet_manager.cc:1397] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260925 14:57:07.882081 5472 raft_consensus.cc:348] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:07.882347 5472 raft_consensus.cc:729] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5c325117754e3482bd9734bf13076b, State: Initialized, Role: FOLLOWER
I20260925 14:57:07.882519 5472 consensus_queue.cc:260] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:07.882776 5472 ts_tablet_manager.cc:1428] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260925 14:57:07.883102 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.882984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:57819)
0925 14:57:07.883036 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.883070 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:57:07.883385 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:07.883292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:56661)
0925 14:57:07.883335 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:07.883358 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:57:07.964700 5584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:07.964964 5584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:07.965008 5584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:07.966322 5584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:07.966391 5584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:07.966439 5584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:07.967965 5584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:33191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=42305
--tserver_master_addrs=127.20.199.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:07.968250 5584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:07.968488 5584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:07.974350 5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:07.974375 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:07.974350 5584 server_base.cc:1034] running on GCE node
W20260925 14:57:07.974375 5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:07.974776 5584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:07.975142 5584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:07.976302 5584 hybrid_clock.cc:648] HybridClock initialized: now 1790348227976276 us; error 41 us; skew 500 ppm
Sep 25 14:57:07 dist-test-slave-8qt6 krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:07.978976 5584 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:57:07.979524 5584 webserver.cc:460] Webserver started at http://127.20.199.3:42305/ using document root <none> and password file <none>
I20260925 14:57:07.979775 5584 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:07.979842 5584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:07.981294 5584 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:07.982050 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:07.982296 5584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:07.982434 5584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6"
format_stamp: "Formatted at 2026-09-25 14:56:26 on dist-test-slave-8qt6"
server_key: "2b5dd4eab3171166941c5ca52c675e93"
server_key_iv: "4b339df5875e3a750c9eedbb8350549c"
server_key_version: "encryptionkey@0"
I20260925 14:57:07.982894 5584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:08.003890 5584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:08.006300 5584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:08.006487 5584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:08.006757 5584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:08.007107 5584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:08.007144 5584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:08.007170 5584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:08.007222 5584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:08.014250 5584 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:33191
I20260925 14:57:08.014312 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:33191 every 8 connection(s)
I20260925 14:57:08.014663 5584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:08.015705 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:08.015331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:40261)
0925 14:57:08.015480 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:08.015553 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260925 14:57:08.016165 5714 heartbeater.cc:644] Failed to heartbeat to 127.20.199.62:45219 (0 consecutive failures): Network error: Failed to ping master at 127.20.199.62:45219: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
I20260925 14:57:08.016475 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:08.016386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:54901)
0925 14:57:08.016431 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:08.016459 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 14:57:08.016744 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:08.016673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:53739)
0925 14:57:08.016711 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:08.016731 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:57:08.016896 5714 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:57:08.023072 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5584
I20260925 14:57:08.802296 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:08.802073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:38335)
0925 14:57:08.802192 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:08.802233 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260925 14:57:08.884080 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:08.883780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:43213)
0925 14:57:08.883973 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:08.884019 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260925 14:57:09.017514 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.017246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:36579)
0925 14:57:09.017363 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.017460 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260925 14:57:09.108966 5721 raft_consensus.cc:482] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:09.109090 5721 raft_consensus.cc:504] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:09.109375 5721 leader_election.cc:288] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191), 5b5c325117754e3482bd9734bf13076b (127.20.199.2:38635)
Sep 25 14:57:09 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.2@KRBTEST.COM
Sep 25 14:57:09 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.3@KRBTEST.COM
I20260925 14:57:09.121037 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.109892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:40293 (local address 127.20.199.2:38635)
0925 14:57:09.110040 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:09.110047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:09.110070 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:57:09.110202 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:09.110204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:09.110256 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:09.110394 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:09.111138 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.112287 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.113371 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.113798 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.113932 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:09.116439 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:09.116443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:09.116443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:09.116454 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:09.118656 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.119308 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.119312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.119312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.119450 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.119871 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.119873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.119873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.120194 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:09.120290 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:09.120763 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 14:57:09.120944 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":50,"threads_started":1}
I20260925 14:57:09.121038 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.109797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:39273 (local address 127.20.199.3:33191)
0925 14:57:09.110028 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:09.110041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:09.110074 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:57:09.110653 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:09.110656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:09.110703 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:09.110845 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:09.111231 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.112284 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.113325 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.113880 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.114010 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:09.116876 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:09.116879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:09.116880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:09.116890 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:09.118909 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.119353 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.119356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.119357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.119450 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:09.119841 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:09.119842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:09.119843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:09.120194 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:09.120290 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:09.120763 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 14:57:09.120910 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
I20260925 14:57:09.121039 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.109649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33191 (local address 127.20.199.1:39273)
0925 14:57:09.109771 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.109787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:09.109907 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:09.110841 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:09.110842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:09.111029 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:09.111110 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:09.111112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.112391 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.112392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:09.113232 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:09.113234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.113981 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.113983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:09.114057 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:09.114634 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:09.114649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:09.116755 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:09.118996 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:09.118997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:09.118997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:09.119241 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:09.119552 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:09.119554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:09.119554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:09.119733 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:09.120309 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:09.120310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:09.120485 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 14:57:09.120657 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 14:57:09.120935 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":148}
I20260925 14:57:09.121039 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.109793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:38635 (local address 127.20.199.1:40293)
0925 14:57:09.109938 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.109949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:09.110001 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:09.110469 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:09.110471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:09.110756 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:09.111020 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:09.111025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.112408 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.112409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:09.113233 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:09.113235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:09.113916 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:09.113917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:09.114054 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:09.114696 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:09.114700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:09.116309 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:09.118757 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:09.118765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:09.118767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:09.119194 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:09.119552 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:09.119554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:09.119554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:09.119733 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:09.120301 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:09.120310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:09.120485 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 14:57:09.120657 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 14:57:09.120880 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":76,"thread_start_us":91,"threads_started":1}
I20260925 14:57:09.121783 5534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37153332b42346bab4505195a35c7af8" candidate_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5b5c325117754e3482bd9734bf13076b" is_pre_election: true
I20260925 14:57:09.121974 5534 raft_consensus.cc:2454] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba34c3b84cc431aa6a5fa4d6428e848 in term 1.
W20260925 14:57:09.122097 5333 leader_election.cc:341] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191): Not found: Tablet not found: 37153332b42346bab4505195a35c7af8
I20260925 14:57:09.122238 5333 leader_election.cc:302] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b5c325117754e3482bd9734bf13076b, dba34c3b84cc431aa6a5fa4d6428e848; no voters: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6
I20260925 14:57:09.122375 5721 raft_consensus.cc:2781] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:57:09.122452 5721 raft_consensus.cc:482] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:09.122486 5721 raft_consensus.cc:3037] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:57:09.123638 5721 raft_consensus.cc:504] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:09.123806 5721 leader_election.cc:288] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [CANDIDATE]: Term 2 election: Requested vote from peers 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191), 5b5c325117754e3482bd9734bf13076b (127.20.199.2:38635)
I20260925 14:57:09.124049 5534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37153332b42346bab4505195a35c7af8" candidate_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5b5c325117754e3482bd9734bf13076b"
W20260925 14:57:09.124135 5333 leader_election.cc:341] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191): Not found: Tablet not found: 37153332b42346bab4505195a35c7af8
I20260925 14:57:09.124155 5534 raft_consensus.cc:3037] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:57:09.125245 5534 raft_consensus.cc:2454] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba34c3b84cc431aa6a5fa4d6428e848 in term 2.
I20260925 14:57:09.125447 5333 leader_election.cc:302] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b5c325117754e3482bd9734bf13076b, dba34c3b84cc431aa6a5fa4d6428e848; no voters: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6
I20260925 14:57:09.125554 5721 raft_consensus.cc:2781] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:57:09.125669 5721 raft_consensus.cc:686] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [term 2 LEADER]: Becoming Leader. State: Replica: dba34c3b84cc431aa6a5fa4d6428e848, State: Running, Role: LEADER
I20260925 14:57:09.125762 5721 consensus_queue.cc:237] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:09.125989 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.125894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:56597)
0925 14:57:09.125929 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.125958 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 14:57:09.659632 5534 raft_consensus.cc:1261] T 37153332b42346bab4505195a35c7af8 P 5b5c325117754e3482bd9734bf13076b [term 2 FOLLOWER]: Refusing update from remote peer dba34c3b84cc431aa6a5fa4d6428e848: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 14:57:09.660120 5727 consensus_queue.cc:1035] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 14:57:09.661233 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.661051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:38309)
0925 14:57:09.661151 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.661197 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
W20260925 14:57:09.663564 5333 consensus_peers.cc:487] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848 -> Peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 (127.20.199.3:33191): Couldn't send request to peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37153332b42346bab4505195a35c7af8. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:57:09.663654 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.663563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:51371)
0925 14:57:09.663600 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.663624 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 14:57:09.663581 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:09.663374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:57345)
0925 14:57:09.663488 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:09.663522 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":43,"threads_started":1}
I20260925 14:57:10.018249 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.018039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:56863)
0925 14:57:10.018170 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.018209 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260925 14:57:10.206022 5736 ts_tablet_manager.cc:927] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Initiating tablet copy from peer dba34c3b84cc431aa6a5fa4d6428e848 (127.20.199.1:34943)
I20260925 14:57:10.206296 5736 tablet_copy_client.cc:316] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.199.1:34943
Sep 25 14:57:10 dist-test-slave-8qt6 krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.1@KRBTEST.COM
I20260925 14:57:10.215137 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34943 (local address 127.20.199.3:36761)
0925 14:57:10.206646 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.206665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:10.206734 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:10.207023 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:10.207028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:10.207313 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:10.207384 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:10.207386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:10.208308 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:10.208309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:10.208901 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:10.208902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:10.209390 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:10.209391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:10.209717 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:10.210310 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:10.210315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:10.211802 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:10.213898 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:10.213905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:10.213906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:10.214129 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:10.214377 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:10.214378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:10.214378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:10.214446 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:10.214802 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:10.214809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:10.214869 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 14:57:10.214945 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:57:10.215044 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:57:10.215173 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.206702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:36761 (local address 127.20.199.1:34943)
0925 14:57:10.206865 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:10.206869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:10.206880 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:10.206905 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:10.206906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:10.206920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:10.206997 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:10.207477 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:10.208217 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:10.208983 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:10.209294 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:10.209358 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:10.211945 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:10.211956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:10.211956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:10.211965 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:10.213757 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:10.214220 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:10.214223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:10.214223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:10.214287 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:10.214536 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:10.214538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:10.214538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:10.214703 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:10.214785 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:10.215034 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:57:10.215105 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":113,"threads_started":1}
I20260925 14:57:10.215647 5419 tablet_copy_service.cc:140] P dba34c3b84cc431aa6a5fa4d6428e848: Received BeginTabletCopySession request for tablet 37153332b42346bab4505195a35c7af8 from peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 ({username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:36761)
I20260925 14:57:10.215709 5419 tablet_copy_service.cc:161] P dba34c3b84cc431aa6a5fa4d6428e848: Beginning new tablet copy session on tablet 37153332b42346bab4505195a35c7af8 from peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 at {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:36761: session id = 00f4ff7ac24440f5bf8cf4ec6dc3c9e6-37153332b42346bab4505195a35c7af8
I20260925 14:57:10.216169 5419 tablet_copy_source_session.cc:205] T 37153332b42346bab4505195a35c7af8 P dba34c3b84cc431aa6a5fa4d6428e848: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:57:10.217110 5736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37153332b42346bab4505195a35c7af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:10.219730 5736 tablet_copy_client.cc:791] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: tablet copy: Starting download of 0 data blocks...
I20260925 14:57:10.219892 5736 tablet_copy_client.cc:655] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: tablet copy: Starting download of 1 WAL segments...
I20260925 14:57:10.220803 5736 tablet_copy_client.cc:531] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:57:10.221566 5736 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Bootstrap starting.
I20260925 14:57:10.223078 5736 log.cc:824] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:10.225562 5736 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:10.225785 5736 tablet_bootstrap.cc:492] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Bootstrap complete.
I20260925 14:57:10.225929 5736 ts_tablet_manager.cc:1397] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 14:57:10.226892 5736 raft_consensus.cc:348] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:10.227066 5736 raft_consensus.cc:729] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f4ff7ac24440f5bf8cf4ec6dc3c9e6, State: Initialized, Role: FOLLOWER
I20260925 14:57:10.227196 5736 consensus_queue.cc:260] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f4ff7ac24440f5bf8cf4ec6dc3c9e6" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33191 } } peers { permanent_uuid: "5b5c325117754e3482bd9734bf13076b" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 38635 } } peers { permanent_uuid: "dba34c3b84cc431aa6a5fa4d6428e848" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34943 } }
I20260925 14:57:10.227449 5736 ts_tablet_manager.cc:1428] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:10.227789 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.227679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:42171)
0925 14:57:10.227737 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.227766 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:57:10.227762 5419 tablet_copy_service.cc:342] P dba34c3b84cc431aa6a5fa4d6428e848: Request end of tablet copy session 00f4ff7ac24440f5bf8cf4ec6dc3c9e6-37153332b42346bab4505195a35c7af8 received from {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:36761
I20260925 14:57:10.227841 5419 tablet_copy_service.cc:433] P dba34c3b84cc431aa6a5fa4d6428e848: ending tablet copy session 00f4ff7ac24440f5bf8cf4ec6dc3c9e6-37153332b42346bab4505195a35c7af8 on tablet 37153332b42346bab4505195a35c7af8 with peer 00f4ff7ac24440f5bf8cf4ec6dc3c9e6
I20260925 14:57:10.228289 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.228196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:34045)
0925 14:57:10.228229 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.228257 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 14:57:10.664438 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.664232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:40273)
0925 14:57:10.664350 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.664394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260925 14:57:10.664451 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.664238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:47267)
0925 14:57:10.664380 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.664411 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260925 14:57:10.707286 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.707154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:47559)
0925 14:57:10.707210 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.707249 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:57:10.707324 5668 raft_consensus.cc:1203] T 37153332b42346bab4505195a35c7af8 P 00f4ff7ac24440f5bf8cf4ec6dc3c9e6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 14:57:10.707883 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:10.707755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:57467)
0925 14:57:10.707819 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:10.707847 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:57:11.665301 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:11.665029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:56233)
0925 14:57:11.665198 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:11.665242 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260925 14:57:11.708220 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:11.707929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:44321)
0925 14:57:11.708122 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:11.708176 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260925 14:57:11.708508 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:11.708290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:44565)
0925 14:57:11.708431 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:11.708466 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":52,"threads_started":1}
I20260925 14:57:12.666045 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:12.665806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:50527)
0925 14:57:12.665958 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:12.665999 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260925 14:57:12.708945 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:12.708684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:59153)
0925 14:57:12.708852 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:12.708898 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260925 14:57:12.709143 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:12.708952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:46195)
0925 14:57:12.709058 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:12.709094 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260925 14:57:13.666917 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:13.666689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:58925)
0925 14:57:13.666823 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:13.666868 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260925 14:57:13.709764 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:13.709527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:51761)
0925 14:57:13.709655 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:13.709699 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260925 14:57:13.709933 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:13.709698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:35075)
0925 14:57:13.709822 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:13.709880 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260925 14:57:13.710135 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:13.710021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:57285)
0925 14:57:13.710073 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:13.710114 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:57:14.667671 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:14.667434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:55741)
0925 14:57:14.667583 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:14.667627 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260925 14:57:14.710703 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:14.710454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:56389)
0925 14:57:14.710611 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:14.710656 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260925 14:57:14.710747 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:14.710534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:51507)
0925 14:57:14.710669 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:14.710704 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 14:57:15.668570 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:15.668307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:41333)
0925 14:57:15.668473 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:15.668522 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 14:57:15.711699 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:15.711337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:48525)
0925 14:57:15.711603 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:15.711653 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":65,"threads_started":1}
I20260925 14:57:15.711709 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:15.711345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:55185)
0925 14:57:15.711631 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:15.711668 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":215,"threads_started":1}
I20260925 14:57:16.669391 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:16.669144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:36795)
0925 14:57:16.669296 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:16.669343 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260925 14:57:16.712591 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:16.712329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:43935)
0925 14:57:16.712502 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:16.712546 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260925 14:57:16.712590 5762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:16.712329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:43509)
0925 14:57:16.712494 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:16.712543 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260925 14:57:17.670222 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:17.670008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:46965)
0925 14:57:17.670144 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:17.670177 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260925 14:57:17.713330 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:17.713113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:43735)
0925 14:57:17.713255 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:17.713285 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260925 14:57:17.713460 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:17.713207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:57889)
0925 14:57:17.713332 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:17.713366 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260925 14:57:18.670936 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:18.670704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:33045)
0925 14:57:18.670847 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:18.670892 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260925 14:57:18.714085 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:18.713847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:49031)
0925 14:57:18.713979 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:18.714040 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260925 14:57:18.714130 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:18.713920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:35235)
0925 14:57:18.714049 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:18.714090 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260925 14:57:19.671703 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:19.671468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:52383)
0925 14:57:19.671614 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:19.671658 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260925 14:57:19.714767 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:19.714540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:34381)
0925 14:57:19.714689 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:19.714722 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260925 14:57:19.714787 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:19.714589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:60885)
0925 14:57:19.714713 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:19.714746 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260925 14:57:20.672453 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:20.672224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:48229)
0925 14:57:20.672359 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:20.672403 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260925 14:57:20.715528 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:20.715303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:55455)
0925 14:57:20.715458 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:20.715488 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260925 14:57:20.715521 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:20.715272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:43901)
0925 14:57:20.715434 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:20.715477 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260925 14:57:21.673197 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:21.672982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:42663)
0925 14:57:21.673111 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:21.673151 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 14:57:21.716197 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:21.715990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:58831)
0925 14:57:21.716120 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:21.716151 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260925 14:57:21.716269 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:21.716035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:60359)
0925 14:57:21.716179 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:21.716209 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260925 14:57:22.673842 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:22.673639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.2:43537)
0925 14:57:22.673761 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:22.673797 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260925 14:57:22.716892 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:22.716685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.1:49261)
0925 14:57:22.716809 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:22.716843 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 14:57:22.716890 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:22.716697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45219 (local address 127.20.199.3:53029)
0925 14:57:22.716811 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:22.716851 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.62:45219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260925 14:57:23.029966 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 5282
I20260925 14:57:23.034853 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 5448
I20260925 14:57:23.039482 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 5584
2026-09-25T14:57:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58267 ms)
[----------] 30 tests from SecurityITest (229013 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:23 dist-test-slave-8qt6 krb5kdc[5786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:23 dist-test-slave-8qt6 krb5kdc[5786](info): set up 2 sockets
Sep 25 14:57:23 dist-test-slave-8qt6 krb5kdc[5786](info): commencing operation
krb5kdc: starting...
W20260925 14:57:24.081164 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:24.200210 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40195
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:40195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:24.280308 5799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:24.280525 5799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:24.280547 5799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:24.281772 5799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:24.281813 5799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:24.281828 5799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:24.281840 5799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:24.281852 5799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:24.283288 5799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:40195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:24.283545 5799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:24.283771 5799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:24.289088 5804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.289134 5807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.289134 5805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:24.289093 5799 server_base.cc:1034] running on GCE node
I20260925 14:57:24.289707 5799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:24.289999 5799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:24.291157 5799 hybrid_clock.cc:648] HybridClock initialized: now 1790348244291148 us; error 29 us; skew 500 ppm
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:24.293793 5799 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:24.294252 5799 webserver.cc:460] Webserver started at http://127.20.199.62:44849/ using document root <none> and password file <none>
I20260925 14:57:24.294481 5799 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:24.294550 5799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:24.294708 5799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:24.295765 5799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "eee41e22450d401ba5eff13e07f48f98"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "5841f8c57a78249ce96c4cd3e937b752"
server_key_iv: "a8fb3ed31439c727867b9c03ebfd808f"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.296142 5799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "eee41e22450d401ba5eff13e07f48f98"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "5841f8c57a78249ce96c4cd3e937b752"
server_key_iv: "a8fb3ed31439c727867b9c03ebfd808f"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.297535 5799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:57:24.298385 5814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.298588 5799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:24.298691 5799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "eee41e22450d401ba5eff13e07f48f98"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "5841f8c57a78249ce96c4cd3e937b752"
server_key_iv: "a8fb3ed31439c727867b9c03ebfd808f"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.298777 5799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:24.312304 5799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:24.314592 5799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:24.314755 5799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:24.318714 5799 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:40195
I20260925 14:57:24.318768 5866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:40195 every 8 connection(s)
I20260925 14:57:24.319129 5799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:24.320057 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:24.322393 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: Bootstrap starting.
I20260925 14:57:24.322973 5867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:24.323262 5867 log.cc:824] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:24.323994 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: No bootstrap required, opened a new log
I20260925 14:57:24.324842 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5799
I20260925 14:57:24.324944 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:24.325133 21276 external_mini_cluster.cc:1428] Setting key 726bd2ef50520eb6c34666f9c31d9d78
I20260925 14:57:24.326185 5867 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee41e22450d401ba5eff13e07f48f98" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40195 } }
I20260925 14:57:24.326341 5867 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:24.326380 5867 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee41e22450d401ba5eff13e07f48f98, State: Initialized, Role: FOLLOWER
I20260925 14:57:24.326505 5867 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee41e22450d401ba5eff13e07f48f98" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40195 } }
I20260925 14:57:24.326587 5867 raft_consensus.cc:388] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:24.326624 5867 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:24.326673 5867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:24.327615 5867 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee41e22450d401ba5eff13e07f48f98" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40195 } }
I20260925 14:57:24.327701 5867 leader_election.cc:302] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eee41e22450d401ba5eff13e07f48f98; no voters:
I20260925 14:57:24.327873 5867 leader_election.cc:288] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:24.327924 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:24.328063 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [term 1 LEADER]: Becoming Leader. State: Replica: eee41e22450d401ba5eff13e07f48f98, State: Running, Role: LEADER
I20260925 14:57:24.328123 5867 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:24.328193 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee41e22450d401ba5eff13e07f48f98" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40195 } }
I20260925 14:57:24.328939 5874 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee41e22450d401ba5eff13e07f48f98. Latest consensus state: current_term: 1 leader_uuid: "eee41e22450d401ba5eff13e07f48f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee41e22450d401ba5eff13e07f48f98" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40195 } } }
I20260925 14:57:24.329025 5874 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:24.329309 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eee41e22450d401ba5eff13e07f48f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee41e22450d401ba5eff13e07f48f98" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40195 } } }
I20260925 14:57:24.329372 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:24.330018 5888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:24.330072 5888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:24.330163 5881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:24.330348 5881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:24.331992 5881 catalog_manager.cc:1347] Generated new cluster ID: a6f626cb6400466893703bba84a5ed76
I20260925 14:57:24.332043 5881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:24.336201 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.325633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.20.199.62:40195)
0925 14:57:24.325818 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.325825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.325850 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:57:24.326168 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.326174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.326218 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.326325 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.327131 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.328268 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.329683 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.330297 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.330395 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.332986 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.332990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.332990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:24.333001 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.334699 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.335071 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.335074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.335075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.335181 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.335407 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.335408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.335409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.335651 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.335818 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.335920 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.336059 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260925 14:57:24.347781 5881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:24.348305 5881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:24.367439 5881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: Generated new TSK 0
I20260925 14:57:24.367662 5881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:57:24.396766 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40195
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:24.503379 5893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:24.503577 5893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:24.503600 5893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:24.504752 5893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:24.504794 5893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:24.504822 5893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:24.506343 5893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:24.506608 5893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:24.506860 5893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:24.511874 5899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.511941 5898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.511927 5901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:24.512084 5893 server_base.cc:1034] running on GCE node
I20260925 14:57:24.512283 5893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:24.512581 5893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:24.513736 5893 hybrid_clock.cc:648] HybridClock initialized: now 1790348244513712 us; error 36 us; skew 500 ppm
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:24.522040 5893 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:57:24.522512 5893 webserver.cc:460] Webserver started at http://127.20.199.1:40985/ using document root <none> and password file <none>
I20260925 14:57:24.522758 5893 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:24.522828 5893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:24.522981 5893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:24.524029 5893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "4066311e5e7f453aa094d53e89c3d90c"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "08dd4d014a1fe9c8355477fb21a9be51"
server_key_iv: "70bfe9450e1c3dfff69f458b1c9fd556"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.524385 5893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "4066311e5e7f453aa094d53e89c3d90c"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "08dd4d014a1fe9c8355477fb21a9be51"
server_key_iv: "70bfe9450e1c3dfff69f458b1c9fd556"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.525730 5893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:24.526446 5908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.526611 5893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:57:24.526710 5893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "4066311e5e7f453aa094d53e89c3d90c"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "08dd4d014a1fe9c8355477fb21a9be51"
server_key_iv: "70bfe9450e1c3dfff69f458b1c9fd556"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.526790 5893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:24.538331 5893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:24.540591 5893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:24.540753 5893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:24.540997 5893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:24.541306 5893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:24.541368 5893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.541460 5893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:24.541505 5893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.547327 5893 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:44233
I20260925 14:57:24.547372 6021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:44233 every 8 connection(s)
I20260925 14:57:24.547685 5893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:24.552587 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 5893
I20260925 14:57:24.552695 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:24.552919 21276 external_mini_cluster.cc:1428] Setting key 22f7672b6035c3e21f7e5dd10b83947b
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:24.557646 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.548361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:55907 (local address 127.20.199.62:40195)
0925 14:57:24.548403 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.548405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.548597 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:57:24.549136 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.549138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.549154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.549205 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.550270 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.550781 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.551769 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.552000 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.552078 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.554571 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.554572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.554573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:24.554583 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.556194 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.556618 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.556619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.556619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.556682 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.556948 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.556949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.556949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.557074 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.557150 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.557460 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.557554 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:57:24.557855 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.548330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40195 (local address 127.20.199.1:55907)
0925 14:57:24.548466 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:24.548510 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:24.549023 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:24.549272 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:24.549276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:24.549973 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:24.550164 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.550169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.550881 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.550882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.551677 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.551679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.552085 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.552086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.552226 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.552780 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:24.552791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:24.554451 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:24.556287 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:24.556295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:24.556297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:24.556537 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:24.556758 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:24.556759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:24.556760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:24.556877 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:24.557152 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:24.557160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:24.557301 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:57:24.557438 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 14:57:24.557598 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260925 14:57:24.558308 6022 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40195
I20260925 14:57:24.558404 6022 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:24.558610 6022 heartbeater.cc:511] Master 127.20.199.62:40195 requested a full tablet report, sending...
I20260925 14:57:24.559125 5831 ts_manager.cc:194] Registered new tserver with Master: 4066311e5e7f453aa094d53e89c3d90c (127.20.199.1:44233)
I20260925 14:57:24.559741 5831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:55907
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:57:24.579891 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40195
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:24.660862 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:24.661064 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:24.661087 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:24.662246 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:24.662288 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:24.662314 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:24.663663 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:24.663843 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:24.664028 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:24.668992 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.669001 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:24.669095 6027 server_base.cc:1034] running on GCE node
W20260925 14:57:24.669001 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:24.669484 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:24.669713 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:24.670856 6027 hybrid_clock.cc:648] HybridClock initialized: now 1790348244670852 us; error 36 us; skew 500 ppm
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:24.673341 6027 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:57:24.673846 6027 webserver.cc:460] Webserver started at http://127.20.199.2:36753/ using document root <none> and password file <none>
I20260925 14:57:24.674090 6027 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:24.674160 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:24.674322 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:24.675359 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "230aa65384724ed39f37e1a689968793"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "dcd5e2b3443c0e8e8dcf35789233c0d8"
server_key_iv: "d748699160f665eb62c31e8794cabb0e"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.675735 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "230aa65384724ed39f37e1a689968793"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "dcd5e2b3443c0e8e8dcf35789233c0d8"
server_key_iv: "d748699160f665eb62c31e8794cabb0e"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.677143 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:24.677970 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.678139 6027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.678237 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "230aa65384724ed39f37e1a689968793"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "dcd5e2b3443c0e8e8dcf35789233c0d8"
server_key_iv: "d748699160f665eb62c31e8794cabb0e"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.678323 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:24.698402 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:24.700750 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:24.700929 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:24.701192 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:24.701553 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:24.701608 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.701650 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:24.701678 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.707623 6027 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:34735
I20260925 14:57:24.707667 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:34735 every 8 connection(s)
I20260925 14:57:24.708029 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:24.714942 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6027
I20260925 14:57:24.715055 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:24.715289 21276 external_mini_cluster.cc:1428] Setting key f6ffc8996e1624a4a7e51f52b819eaf2
I20260925 14:57:24.717772 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.708818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:53331 (local address 127.20.199.62:40195)
0925 14:57:24.708898 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.708900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.709130 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:57:24.709673 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.709674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.709684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.709743 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.710630 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.711126 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.712170 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.712352 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.712410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.714668 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.714669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.714669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:24.714676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.716258 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.716723 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.716723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.716724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.716773 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.717055 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.717055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.717055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.717150 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.717199 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.717567 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.717636 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:57:24.718024 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.708803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40195 (local address 127.20.199.2:53331)
0925 14:57:24.708983 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:24.709037 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:24.709588 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:24.709796 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:24.709800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:24.710346 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:24.710538 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.710543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.711241 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.711242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.712067 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.712070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.712452 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.712453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.712568 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.713034 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:24.713042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:24.714579 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:24.716317 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:24.716324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:24.716326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:24.716647 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:24.716845 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:24.716846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:24.716846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:24.716992 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:24.717212 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:24.717218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:24.717387 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 14:57:24.717537 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 14:57:24.717723 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260925 14:57:24.718477 6156 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40195
I20260925 14:57:24.718586 6156 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:24.718796 6156 heartbeater.cc:511] Master 127.20.199.62:40195 requested a full tablet report, sending...
I20260925 14:57:24.719203 5831 ts_manager.cc:194] Registered new tserver with Master: 230aa65384724ed39f37e1a689968793 (127.20.199.2:34735)
I20260925 14:57:24.719587 5831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:53331
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:57:24.740348 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40195
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:24.822384 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:24.822598 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:24.822623 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:24.823817 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:24.823879 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:24.823923 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:24.825549 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:24.825788 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:24.826047 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:24.831557 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.831555 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:24.831553 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:24.831868 6161 server_base.cc:1034] running on GCE node
I20260925 14:57:24.832113 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:24.832376 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:24.833467 6161 hybrid_clock.cc:648] HybridClock initialized: now 1790348244833456 us; error 30 us; skew 500 ppm
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:24.836109 6161 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:57:24.836609 6161 webserver.cc:460] Webserver started at http://127.20.199.3:44211/ using document root <none> and password file <none>
I20260925 14:57:24.836840 6161 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:24.836908 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:24.837046 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:24.838177 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "3e5b34ecb08b43679163d1e3f1cdde10"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "ef3059ff3befa560e1f87d6c050eb6fa"
server_key_iv: "4f8dd6fd306b1578acd22f232c381bd1"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.838544 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "3e5b34ecb08b43679163d1e3f1cdde10"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "ef3059ff3befa560e1f87d6c050eb6fa"
server_key_iv: "4f8dd6fd306b1578acd22f232c381bd1"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.839862 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:57:24.840646 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.840826 6161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:57:24.840934 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "3e5b34ecb08b43679163d1e3f1cdde10"
format_stamp: "Formatted at 2026-09-25 14:57:24 on dist-test-slave-8qt6"
server_key: "ef3059ff3befa560e1f87d6c050eb6fa"
server_key_iv: "4f8dd6fd306b1578acd22f232c381bd1"
server_key_version: "encryptionkey@0"
I20260925 14:57:24.841020 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:24.853590 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:24.855923 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:24.856087 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:24.856319 6161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:24.856635 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:24.856690 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.856732 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:24.856760 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:24.862563 6161 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:40743
I20260925 14:57:24.862608 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:40743 every 8 connection(s)
I20260925 14:57:24.862917 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:24.865134 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6161
I20260925 14:57:24.865223 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:24.865530 21276 external_mini_cluster.cc:1428] Setting key c51a73d511c58f4acbd257462f249cd0
Sep 25 14:57:24 dist-test-slave-8qt6 krb5kdc[5786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348244, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:24.872324 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.863574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:54345 (local address 127.20.199.62:40195)
0925 14:57:24.863620 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.863622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.863871 (+ 249us) server_negotiation.cc:407] Connection header received
0925 14:57:24.864400 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.864401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.864413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.864480 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.865574 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.866326 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.867299 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.867515 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.867606 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.869729 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.869730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.869730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:24.869737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.870962 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.871374 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.871375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.871376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.871434 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.871684 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.871685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.871685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.871798 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.871870 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.872157 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.872234 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:24.872505 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.863550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:40195 (local address 127.20.199.3:54345)
0925 14:57:24.863722 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:24.863780 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:24.864291 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:24.864551 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:24.864555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:24.865151 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:24.865472 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.865478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.866401 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.866402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.867203 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.867206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.867616 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.867618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.867751 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.868249 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:24.868257 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:24.869629 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:24.871059 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:24.871066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:24.871068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:24.871283 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:24.871502 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:24.871503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:24.871504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:24.871615 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:24.871872 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:24.871880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:24.872022 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 14:57:24.872123 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:57:24.872281 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260925 14:57:24.873039 6290 heartbeater.cc:347] Connected to a master server at 127.20.199.62:40195
I20260925 14:57:24.873134 6290 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:24.873304 6290 heartbeater.cc:511] Master 127.20.199.62:40195 requested a full tablet report, sending...
I20260925 14:57:24.873714 5831 ts_manager.cc:194] Registered new tserver with Master: 3e5b34ecb08b43679163d1e3f1cdde10 (127.20.199.3:40743)
I20260925 14:57:24.874073 5831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:54345
I20260925 14:57:24.877290 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:24.884646 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.878636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.20.199.62:40195)
0925 14:57:24.878724 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.878726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.878826 (+ 100us) server_negotiation.cc:407] Connection header received
0925 14:57:24.878904 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.878905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.878915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.878970 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:24.879525 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.880003 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.880749 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.880986 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.881062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.882299 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:24.882301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:24.882301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:24.882308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:24.883457 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.883804 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.883805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.883805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.883894 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:24.884096 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:24.884097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:24.884098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:24.884244 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:24.884351 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.884457 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.884535 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:24.886408 5831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:24.888085 5831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:24.894518 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.890416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:34735 (local address 127.0.0.1:49450)
0925 14:57:24.890582 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:24.890625 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:24.890696 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:24.891219 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:24.891220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:24.891222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:24.891286 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.891287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.892198 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.892199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.893233 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.893235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.894199 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.894285 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.894300 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:57:24.894388 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:57:24.894439 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260925 14:57:24.894531 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.890506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.20.199.2:34735)
0925 14:57:24.890859 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.890864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.890875 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:24.890918 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.890920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.890936 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.891015 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:24.891376 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.892089 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.893331 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.894116 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894184 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.894240 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.894411 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.894452 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":78,"threads_started":1}
I20260925 14:57:24.895133 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.890644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:40743 (local address 127.0.0.1:60250)
0925 14:57:24.890869 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:24.890880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:24.890929 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:24.891305 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:24.891305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:24.891307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:24.891339 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.891340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.892796 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.892797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.893929 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.893932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.894870 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.894935 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.894939 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:24.895021 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:57:24.895064 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260925 14:57:24.895188 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.890856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.20.199.3:40743)
0925 14:57:24.891043 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.891057 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.891066 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:24.891098 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.891100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.891138 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.891222 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:24.891423 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.892662 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894033 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.894772 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894857 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.894911 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.895066 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.895105 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260925 14:57:24.895285 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.890435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:44233 (local address 127.0.0.1:47104)
0925 14:57:24.890774 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:24.890785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:24.890844 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:24.891169 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:24.891172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:24.891174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:24.891326 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.891328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.892798 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.892799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.893868 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:24.893871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.894957 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:24.895030 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.895035 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:57:24.895090 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:57:24.895134 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
I20260925 14:57:24.895314 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:24.890626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.20.199.1:44233)
0925 14:57:24.890995 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:24.890999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:24.891009 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:24.891044 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:24.891045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:24.891058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:24.891134 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:24.891425 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.892667 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.893984 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:24.894780 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:24.894862 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:24.894919 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:24.895178 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 14:57:24.895223 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":102,"threads_started":1}
I20260925 14:57:24.895659 6090 tablet_service.cc:1467] Processing CreateTablet for tablet 9babf1b64d004f77b42c7024ff5086bc (DEFAULT_TABLE table=test-table [id=6dbcb711577d4767993334614ffb34bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:24.895975 6090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9babf1b64d004f77b42c7024ff5086bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:24.898077 6224 tablet_service.cc:1467] Processing CreateTablet for tablet 9babf1b64d004f77b42c7024ff5086bc (DEFAULT_TABLE table=test-table [id=6dbcb711577d4767993334614ffb34bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:24.898416 6224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9babf1b64d004f77b42c7024ff5086bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:24.898859 6307 tablet_bootstrap.cc:492] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Bootstrap starting.
I20260925 14:57:24.899489 6307 tablet_bootstrap.cc:654] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:24.900048 6307 log.cc:824] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:24.900751 6307 tablet_bootstrap.cc:492] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: No bootstrap required, opened a new log
I20260925 14:57:24.900817 6307 ts_tablet_manager.cc:1397] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:24.901517 5956 tablet_service.cc:1467] Processing CreateTablet for tablet 9babf1b64d004f77b42c7024ff5086bc (DEFAULT_TABLE table=test-table [id=6dbcb711577d4767993334614ffb34bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:24.901588 6309 tablet_bootstrap.cc:492] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Bootstrap starting.
I20260925 14:57:24.901808 5956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9babf1b64d004f77b42c7024ff5086bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:24.902351 6309 tablet_bootstrap.cc:654] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:24.903021 6309 log.cc:824] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:24.903617 6307 raft_consensus.cc:348] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:24.903842 6309 tablet_bootstrap.cc:492] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: No bootstrap required, opened a new log
I20260925 14:57:24.903831 6307 raft_consensus.cc:374] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:24.903882 6307 raft_consensus.cc:729] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230aa65384724ed39f37e1a689968793, State: Initialized, Role: FOLLOWER
I20260925 14:57:24.903901 6309 ts_tablet_manager.cc:1397] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:24.904072 6307 consensus_queue.cc:260] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:24.904374 6307 ts_tablet_manager.cc:1428] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:24.904505 6156 heartbeater.cc:503] Master 127.20.199.62:40195 was elected leader, sending a full tablet report...
I20260925 14:57:24.904670 6312 tablet_bootstrap.cc:492] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: Bootstrap starting.
I20260925 14:57:24.905272 6312 tablet_bootstrap.cc:654] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:24.905928 6312 log.cc:824] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:24.906116 6309 raft_consensus.cc:348] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:24.906327 6309 raft_consensus.cc:374] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:24.906383 6309 raft_consensus.cc:729] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5b34ecb08b43679163d1e3f1cdde10, State: Initialized, Role: FOLLOWER
I20260925 14:57:24.906533 6309 consensus_queue.cc:260] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:24.906752 6312 tablet_bootstrap.cc:492] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: No bootstrap required, opened a new log
I20260925 14:57:24.906826 6309 ts_tablet_manager.cc:1428] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:57:24.906838 6312 ts_tablet_manager.cc:1397] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:24.906829 6290 heartbeater.cc:503] Master 127.20.199.62:40195 was elected leader, sending a full tablet report...
I20260925 14:57:24.908617 6312 raft_consensus.cc:348] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:24.908740 6312 raft_consensus.cc:374] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:24.908762 6312 raft_consensus.cc:729] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4066311e5e7f453aa094d53e89c3d90c, State: Initialized, Role: FOLLOWER
I20260925 14:57:24.908852 6312 consensus_queue.cc:260] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:24.909085 6312 ts_tablet_manager.cc:1428] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:24.909116 6022 heartbeater.cc:503] Master 127.20.199.62:40195 was elected leader, sending a full tablet report...
W20260925 14:57:24.958669 6157 tablet.cc:2369] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:25.020244 6314 raft_consensus.cc:482] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:25.020417 6314 raft_consensus.cc:504] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:25.020833 6314 leader_election.cc:288] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 230aa65384724ed39f37e1a689968793 (127.20.199.2:34735), 4066311e5e7f453aa094d53e89c3d90c (127.20.199.1:44233)
I20260925 14:57:25.023610 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.021102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:40135 (local address 127.20.199.2:34735)
0925 14:57:25.021149 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.021151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.021159 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:25.021189 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.021190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.021201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.021258 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.021442 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.021910 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.022861 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.023240 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.023326 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.023334 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.023507 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.023545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:57:25.023610 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.020983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:34735 (local address 127.20.199.3:40135)
0925 14:57:25.021043 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.021055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.021101 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.021263 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.021264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.021266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.021342 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.021344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.022028 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.022029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.022762 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.022764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.023334 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.023334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.023412 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.023415 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.023495 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.023538 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:57:25.023934 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.021144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:34095 (local address 127.20.199.1:44233)
0925 14:57:25.021202 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.021204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.021296 (+ 92us) server_negotiation.cc:407] Connection header received
0925 14:57:25.021378 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.021379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.021392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.021516 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.021682 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.022175 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.023140 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.023538 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.023627 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.023634 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.023825 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.023868 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:57:25.023962 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babf1b64d004f77b42c7024ff5086bc" candidate_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230aa65384724ed39f37e1a689968793" is_pre_election: true
I20260925 14:57:25.024144 6110 raft_consensus.cc:2454] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5b34ecb08b43679163d1e3f1cdde10 in term 0.
I20260925 14:57:25.024187 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.021006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:44233 (local address 127.20.199.3:34095)
0925 14:57:25.021193 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.021204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.021312 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.021520 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.021521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.021522 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.021601 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.021603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.022285 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.022286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.023041 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.023042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.023639 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.023640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.023716 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.023721 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.024053 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.024104 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260925 14:57:25.024405 6177 leader_election.cc:302] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230aa65384724ed39f37e1a689968793, 3e5b34ecb08b43679163d1e3f1cdde10; no voters:
I20260925 14:57:25.024562 5976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babf1b64d004f77b42c7024ff5086bc" candidate_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4066311e5e7f453aa094d53e89c3d90c" is_pre_election: true
I20260925 14:57:25.024592 6314 raft_consensus.cc:2781] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:25.024685 6314 raft_consensus.cc:482] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:25.024717 5976 raft_consensus.cc:2454] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5b34ecb08b43679163d1e3f1cdde10 in term 0.
I20260925 14:57:25.024730 6314 raft_consensus.cc:3037] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.025604 6314 raft_consensus.cc:504] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:25.025759 6314 leader_election.cc:288] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [CANDIDATE]: Term 1 election: Requested vote from peers 230aa65384724ed39f37e1a689968793 (127.20.199.2:34735), 4066311e5e7f453aa094d53e89c3d90c (127.20.199.1:44233)
I20260925 14:57:25.025890 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babf1b64d004f77b42c7024ff5086bc" candidate_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230aa65384724ed39f37e1a689968793"
I20260925 14:57:25.025914 5976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babf1b64d004f77b42c7024ff5086bc" candidate_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4066311e5e7f453aa094d53e89c3d90c"
I20260925 14:57:25.025986 5976 raft_consensus.cc:3037] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.025995 6110 raft_consensus.cc:3037] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.026739 5976 raft_consensus.cc:2454] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5b34ecb08b43679163d1e3f1cdde10 in term 1.
I20260925 14:57:25.026899 6110 raft_consensus.cc:2454] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5b34ecb08b43679163d1e3f1cdde10 in term 1.
I20260925 14:57:25.026932 6179 leader_election.cc:302] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e5b34ecb08b43679163d1e3f1cdde10, 4066311e5e7f453aa094d53e89c3d90c; no voters:
I20260925 14:57:25.027034 6314 raft_consensus.cc:2781] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:25.027194 6314 raft_consensus.cc:686] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 1 LEADER]: Becoming Leader. State: Replica: 3e5b34ecb08b43679163d1e3f1cdde10, State: Running, Role: LEADER
I20260925 14:57:25.027302 6314 consensus_queue.cc:237] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } }
I20260925 14:57:25.028177 5831 catalog_manager.cc:5495] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e5b34ecb08b43679163d1e3f1cdde10 (127.20.199.3). New cstate: current_term: 1 leader_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e5b34ecb08b43679163d1e3f1cdde10" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 40743 } health_report { overall_health: HEALTHY } } }
I20260925 14:57:25.039352 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.036877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.20.199.3:40743)
0925 14:57:25.036925 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.036927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.036936 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:25.036979 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.036980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.036997 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.037069 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:25.037230 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.037768 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.038549 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.038741 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.038817 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.039013 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:25.039113 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:25.039206 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.039226 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.039245 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:25.041272 5976 raft_consensus.cc:1261] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 1 FOLLOWER]: Refusing update from remote peer 3e5b34ecb08b43679163d1e3f1cdde10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:25.041385 6110 raft_consensus.cc:1261] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 1 FOLLOWER]: Refusing update from remote peer 3e5b34ecb08b43679163d1e3f1cdde10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:25.041639 6317 consensus_queue.cc:1035] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4066311e5e7f453aa094d53e89c3d90c" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:25.041803 6314 consensus_queue.cc:1035] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "230aa65384724ed39f37e1a689968793" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:25.044293 6322 mvcc.cc:205] Tried to move back new op lower bound from 7333266411686215680 to 7333266411632680960. Current Snapshot: MvccSnapshot[applied={T|T < 7333266411686215680}]
I20260925 14:57:25.044888 6321 mvcc.cc:205] Tried to move back new op lower bound from 7333266411686215680 to 7333266411632680960. Current Snapshot: MvccSnapshot[applied={T|T < 7333266411686215680}]
I20260925 14:57:25.047601 5830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:25.047703 5830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:25.048484 5830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eee41e22450d401ba5eff13e07f48f98: Sending DeleteTablet for 3 replicas of tablet 9babf1b64d004f77b42c7024ff5086bc
I20260925 14:57:25.048871 5956 tablet_service.cc:1514] Processing DeleteTablet for tablet 9babf1b64d004f77b42c7024ff5086bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:25 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:47104
I20260925 14:57:25.048872 6090 tablet_service.cc:1514] Processing DeleteTablet for tablet 9babf1b64d004f77b42c7024ff5086bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:25 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:49450
I20260925 14:57:25.049131 6330 tablet_replica.cc:330] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c: stopping tablet replica
I20260925 14:57:25.049168 6224 tablet_service.cc:1514] Processing DeleteTablet for tablet 9babf1b64d004f77b42c7024ff5086bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:25 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:60250
I20260925 14:57:25.049232 6330 raft_consensus.cc:2229] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:25.049353 6330 raft_consensus.cc:2258] T 9babf1b64d004f77b42c7024ff5086bc P 4066311e5e7f453aa094d53e89c3d90c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:25.049576 6331 tablet_replica.cc:330] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: stopping tablet replica
I20260925 14:57:25.049571 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 5893
I20260925 14:57:25.050305 6331 raft_consensus.cc:2229] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:25.050420 6331 raft_consensus.cc:2258] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:25.050891 6331 ts_tablet_manager.cc:1905] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:25.052515 6332 tablet_replica.cc:330] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: stopping tablet replica
I20260925 14:57:25.052613 6332 raft_consensus.cc:2229] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:25.052623 6331 ts_tablet_manager.cc:1918] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:25.052736 6332 raft_consensus.cc:2258] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:25.052749 6331 log.cc:1196] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/9babf1b64d004f77b42c7024ff5086bc
I20260925 14:57:25.053164 6331 ts_tablet_manager.cc:1939] T 9babf1b64d004f77b42c7024ff5086bc P 230aa65384724ed39f37e1a689968793: Deleting consensus metadata
I20260925 14:57:25.053174 6332 ts_tablet_manager.cc:1905] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:25.053678 5815 catalog_manager.cc:4841] TS 230aa65384724ed39f37e1a689968793 (127.20.199.2:34735): tablet 9babf1b64d004f77b42c7024ff5086bc (table test-table [id=6dbcb711577d4767993334614ffb34bf]) successfully deleted
I20260925 14:57:25.054862 6332 ts_tablet_manager.cc:1918] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:25.054939 6332 log.cc:1196] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/9babf1b64d004f77b42c7024ff5086bc
I20260925 14:57:25.055207 6332 ts_tablet_manager.cc:1939] T 9babf1b64d004f77b42c7024ff5086bc P 3e5b34ecb08b43679163d1e3f1cdde10: Deleting consensus metadata
I20260925 14:57:25.055773 5817 catalog_manager.cc:4841] TS 3e5b34ecb08b43679163d1e3f1cdde10 (127.20.199.3:40743): tablet 9babf1b64d004f77b42c7024ff5086bc (table test-table [id=6dbcb711577d4767993334614ffb34bf]) successfully deleted
W20260925 14:57:25.055918 5817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:44233 (error 108)
I20260925 14:57:25.056146 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6027
I20260925 14:57:25.056762 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.056305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:44233 (local address 127.0.0.1:47120)
0925 14:57:25.056680 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.056721 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260925 14:57:25.057156 5817 catalog_manager.cc:4584] TS 4066311e5e7f453aa094d53e89c3d90c (127.20.199.1:44233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9babf1b64d004f77b42c7024ff5086bc: Network error: Client connection negotiation failed: client connection to 127.20.199.1:44233: connect: Connection refused (error 111)
I20260925 14:57:25.061029 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6161
I20260925 14:57:25.064822 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 5799
2026-09-25T14:57:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2024 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T14:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:25Z Disabled control of system clock
I20260925 14:57:25.092497 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37495
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:44769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:37495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:25.174252 6337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:25.174471 6337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:25.174494 6337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:25.175664 6337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:25.175711 6337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:25.175724 6337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:25.175736 6337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:25.175786 6337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:25.177155 6337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:37495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:25.177480 6337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:25.177788 6337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:25.182525 6342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:25.182492 6343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:25.182674 6337 server_base.cc:1034] running on GCE node
W20260925 14:57:25.182498 6345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:25.183032 6337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:25.183357 6337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:25.184516 6337 hybrid_clock.cc:648] HybridClock initialized: now 1790348245184503 us; error 30 us; skew 500 ppm
I20260925 14:57:25.185714 6337 webserver.cc:460] Webserver started at http://127.20.199.62:36523/ using document root <none> and password file <none>
I20260925 14:57:25.186014 6337 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:25.186084 6337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:25.186218 6337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:25.187242 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "09c7c093769aca1686e841af60643ae2"
server_key_iv: "fdc7d4bd23cdc31fa3a55156c2097a0d"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.187603 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "09c7c093769aca1686e841af60643ae2"
server_key_iv: "fdc7d4bd23cdc31fa3a55156c2097a0d"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.188899 6337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:25.189659 6351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.189853 6337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:25.189951 6337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "09c7c093769aca1686e841af60643ae2"
server_key_iv: "fdc7d4bd23cdc31fa3a55156c2097a0d"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.190033 6337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:25.203012 6337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:25.203333 6337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:25.203486 6337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:25.207533 6337 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:37495
I20260925 14:57:25.207566 6403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:37495 every 8 connection(s)
I20260925 14:57:25.207921 6337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:25.208860 6404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:25.211269 6404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba: Bootstrap starting.
I20260925 14:57:25.211845 6404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:25.212153 6404 log.cc:824] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:25.212867 6404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba: No bootstrap required, opened a new log
I20260925 14:57:25.214534 6404 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37495 } }
I20260925 14:57:25.214680 6404 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:25.214704 6404 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0fc876ec1f54f68a38e5cc2a10ac7ba, State: Initialized, Role: FOLLOWER
I20260925 14:57:25.214864 6404 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37495 } }
I20260925 14:57:25.214928 6404 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:25.214949 6404 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:25.215009 6404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.215570 6404 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37495 } }
I20260925 14:57:25.215656 6404 leader_election.cc:302] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0fc876ec1f54f68a38e5cc2a10ac7ba; no voters:
I20260925 14:57:25.215837 6404 leader_election.cc:288] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:25.215893 6407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:25.216106 6404 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:25.216197 6407 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [term 1 LEADER]: Becoming Leader. State: Replica: b0fc876ec1f54f68a38e5cc2a10ac7ba, State: Running, Role: LEADER
I20260925 14:57:25.216332 6407 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37495 } }
I20260925 14:57:25.216698 6407 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0fc876ec1f54f68a38e5cc2a10ac7ba. Latest consensus state: current_term: 1 leader_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37495 } } }
I20260925 14:57:25.216786 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6337
I20260925 14:57:25.216879 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:25.216836 6408 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fc876ec1f54f68a38e5cc2a10ac7ba" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37495 } } }
I20260925 14:57:25.216969 6408 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:25.217095 21276 external_mini_cluster.cc:1428] Setting key 23edeab95cb0e03cacc26b854a4e10c8
I20260925 14:57:25.217206 6407 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:25.217231 6416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:25.218196 6416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:25.219700 6416 catalog_manager.cc:1347] Generated new cluster ID: 0d7d47a419fb487aba6d5396753b2faf
I20260925 14:57:25.219758 6416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:25.223222 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.217634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.20.199.62:37495)
0925 14:57:25.217836 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.217845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.217878 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:57:25.218105 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.218115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.218193 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.218370 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:25.218601 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.219936 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.221041 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.221661 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.221779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.222513 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:25.222517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:25.222517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:25.222727 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:25.222766 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:25.222779 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:25.222849 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.222949 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.223024 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260925 14:57:25.226652 6416 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:25.227135 6416 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:25.233371 6416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba: Generated new TSK 0
I20260925 14:57:25.233588 6416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:25.236343 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37495
--builtin_ntp_servers=127.20.199.20:44769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:25.320302 6428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:25.320536 6428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:25.320560 6428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:25.321902 6428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:25.321954 6428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:25.321981 6428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:25.323551 6428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:25.323820 6428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:25.324172 6428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:25.329340 6433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:25.329362 6434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:25.329597 6428 server_base.cc:1034] running on GCE node
W20260925 14:57:25.329376 6436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:25.329924 6428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:25.330212 6428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:25.331396 6428 hybrid_clock.cc:648] HybridClock initialized: now 1790348245331377 us; error 30 us; skew 500 ppm
I20260925 14:57:25.332796 6428 webserver.cc:460] Webserver started at http://127.20.199.1:42437/ using document root <none> and password file <none>
I20260925 14:57:25.333065 6428 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:25.333141 6428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:25.333282 6428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:25.334492 6428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "3b809a02d6b443818b374e2d33baee66"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "ec65054424ab5c546cab08daea01c27b"
server_key_iv: "91d9a1c3f9c22ee7559c17e874128d4f"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.334874 6428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "3b809a02d6b443818b374e2d33baee66"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "ec65054424ab5c546cab08daea01c27b"
server_key_iv: "91d9a1c3f9c22ee7559c17e874128d4f"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.336277 6428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:57:25.337088 6442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.337343 6428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:25.337499 6428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "3b809a02d6b443818b374e2d33baee66"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "ec65054424ab5c546cab08daea01c27b"
server_key_iv: "91d9a1c3f9c22ee7559c17e874128d4f"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.337610 6428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:25.355916 6428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:25.356313 6428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:25.356498 6428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:25.356801 6428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:25.357197 6428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:25.357263 6428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.357353 6428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:25.357393 6428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.364245 6428 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:34677
I20260925 14:57:25.364292 6555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:34677 every 8 connection(s)
I20260925 14:57:25.364720 6428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:25.371142 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.365622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:45211 (local address 127.20.199.62:37495)
0925 14:57:25.365703 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.365706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.365992 (+ 286us) server_negotiation.cc:407] Connection header received
0925 14:57:25.366635 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.366637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.366654 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.366738 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:25.367550 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.368129 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.369294 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.369658 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.369762 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.370777 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:25.370778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:25.370779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:25.370785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:25.370802 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:25.370807 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:25.370900 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.371012 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.371049 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:57:25.371384 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.365631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37495 (local address 127.20.199.1:45211)
0925 14:57:25.365858 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.365895 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.366507 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.366879 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.366883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.366891 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:25.367434 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.367439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.368257 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.368259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.369155 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.369158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.369772 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.369774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.369932 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.370591 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:25.370592 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:25.370623 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:25.370648 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:25.370904 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:25.370910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:25.370914 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.371016 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.371138 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260925 14:57:25.371635 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6428
I20260925 14:57:25.371768 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:25.372058 21276 external_mini_cluster.cc:1428] Setting key c64f2f6e0e81767e468122f0c02be851
I20260925 14:57:25.372087 6556 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37495
I20260925 14:57:25.372185 6556 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:25.372422 6556 heartbeater.cc:511] Master 127.20.199.62:37495 requested a full tablet report, sending...
I20260925 14:57:25.373086 6368 ts_manager.cc:194] Registered new tserver with Master: 3b809a02d6b443818b374e2d33baee66 (127.20.199.1:34677)
I20260925 14:57:25.373242 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37495
--builtin_ntp_servers=127.20.199.20:44769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:25.373906 6368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.1:45211
W20260925 14:57:25.457819 6559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:25.458055 6559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:25.458096 6559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:25.459226 6559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:25.459295 6559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:25.459347 6559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:25.460695 6559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:25.460963 6559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:25.461259 6559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:25.466394 6564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:25.466361 6565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:25.466396 6567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:25.466732 6559 server_base.cc:1034] running on GCE node
I20260925 14:57:25.466923 6559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:25.467121 6559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:25.468263 6559 hybrid_clock.cc:648] HybridClock initialized: now 1790348245468248 us; error 31 us; skew 500 ppm
I20260925 14:57:25.469496 6559 webserver.cc:460] Webserver started at http://127.20.199.2:44431/ using document root <none> and password file <none>
I20260925 14:57:25.469735 6559 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:25.469779 6559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:25.469872 6559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:25.470880 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "6a5219a80eab4fd19df6806b26b4c39e"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "772d7a6fb1eec38f7f017f9bd79cdc18"
server_key_iv: "544252842fd84d14e9b51e4272ae6e4d"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.471194 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "6a5219a80eab4fd19df6806b26b4c39e"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "772d7a6fb1eec38f7f017f9bd79cdc18"
server_key_iv: "544252842fd84d14e9b51e4272ae6e4d"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.472393 6559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:25.473083 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.473294 6559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:25.473387 6559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "6a5219a80eab4fd19df6806b26b4c39e"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "772d7a6fb1eec38f7f017f9bd79cdc18"
server_key_iv: "544252842fd84d14e9b51e4272ae6e4d"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.473505 6559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:25.488821 6559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:25.489121 6559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:25.489228 6559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:25.489476 6559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:25.489837 6559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:25.489878 6559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.489934 6559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:25.489969 6559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.495797 6559 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:45629
I20260925 14:57:25.495858 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:45629 every 8 connection(s)
I20260925 14:57:25.496232 6559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:25.498462 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6559
I20260925 14:57:25.498572 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:25.498793 21276 external_mini_cluster.cc:1428] Setting key 5d0750459bc4e9a5552b55b1fdb6f632
I20260925 14:57:25.499893 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37495
--builtin_ntp_servers=127.20.199.20:44769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:25.502136 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.496992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:50775 (local address 127.20.199.62:37495)
0925 14:57:25.497044 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.497046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.497303 (+ 257us) server_negotiation.cc:407] Connection header received
0925 14:57:25.497954 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.497956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.497971 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.498023 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:25.498936 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.499561 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.500667 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.500862 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.500943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.501845 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:25.501846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:25.501847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:25.501853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:25.501867 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:25.501872 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:25.501925 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.502028 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.502059 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:25.502363 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.496971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37495 (local address 127.20.199.2:50775)
0925 14:57:25.497173 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.497209 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.497826 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.498099 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.498103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.498110 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:25.498796 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.498802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.499654 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.499655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.500568 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.500571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.500950 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.500951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.501094 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.501701 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:25.501701 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:25.501731 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:25.501755 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:25.501931 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:25.501935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:25.501938 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.502030 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.502137 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260925 14:57:25.502868 6687 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37495
I20260925 14:57:25.502985 6687 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:25.503170 6687 heartbeater.cc:511] Master 127.20.199.62:37495 requested a full tablet report, sending...
I20260925 14:57:25.503579 6368 ts_manager.cc:194] Registered new tserver with Master: 6a5219a80eab4fd19df6806b26b4c39e (127.20.199.2:45629)
I20260925 14:57:25.503948 6368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.2:50775
W20260925 14:57:25.581601 6690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:25.581808 6690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:25.581832 6690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:25.583030 6690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:25.583076 6690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:25.583103 6690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:25.584586 6690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:25.584875 6690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:25.585129 6690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:25.590152 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:25.590201 6698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:25.590319 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:25.590366 6690 server_base.cc:1034] running on GCE node
I20260925 14:57:25.590590 6690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:25.590831 6690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:25.592003 6690 hybrid_clock.cc:648] HybridClock initialized: now 1790348245591974 us; error 39 us; skew 500 ppm
I20260925 14:57:25.593565 6690 webserver.cc:460] Webserver started at http://127.20.199.3:38679/ using document root <none> and password file <none>
I20260925 14:57:25.593844 6690 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:25.593904 6690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:25.594013 6690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:25.595185 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "e394e0ed11e145cf89fd38d597a6c79f"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "266172c22357792d2c5652b571e675fc"
server_key_iv: "466f6ddbac6d335391fdafa0d97825f4"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.595534 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "e394e0ed11e145cf89fd38d597a6c79f"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "266172c22357792d2c5652b571e675fc"
server_key_iv: "466f6ddbac6d335391fdafa0d97825f4"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.597023 6690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:25.597926 6704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.598131 6690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:25.598230 6690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "e394e0ed11e145cf89fd38d597a6c79f"
format_stamp: "Formatted at 2026-09-25 14:57:25 on dist-test-slave-8qt6"
server_key: "266172c22357792d2c5652b571e675fc"
server_key_iv: "466f6ddbac6d335391fdafa0d97825f4"
server_key_version: "encryptionkey@0"
I20260925 14:57:25.598291 6690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:25.621942 6690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:25.622257 6690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:25.622377 6690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:25.622589 6690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:25.622901 6690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:25.622936 6690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.622962 6690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:25.622977 6690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:25.629160 6690 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:38821
I20260925 14:57:25.629266 6817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:38821 every 8 connection(s)
I20260925 14:57:25.629662 6690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:25.635080 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.630537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:41469 (local address 127.20.199.62:37495)
0925 14:57:25.630627 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.630629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.630814 (+ 185us) server_negotiation.cc:407] Connection header received
0925 14:57:25.631438 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.631439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.631451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.631517 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:25.632158 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.632662 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.633663 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.633849 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.633913 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.634841 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:25.634841 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:25.634842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:25.634845 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:25.634857 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:25.634860 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:25.634900 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.634995 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.635019 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:57:25.635288 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.630486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37495 (local address 127.20.199.3:41469)
0925 14:57:25.630682 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.630725 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.631339 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.631574 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.631578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.631586 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:25.632083 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.632088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.632751 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.632752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.633587 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.633589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.633964 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.633965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.634089 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.634712 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:25.634712 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:25.634742 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:25.634766 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:25.634929 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:25.634932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:25.634934 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.634987 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.635069 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260925 14:57:25.635416 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6690
I20260925 14:57:25.635514 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:25.635705 21276 external_mini_cluster.cc:1428] Setting key 0c4b58e8097d5307067c789f5bcc5fd6
I20260925 14:57:25.635917 6818 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37495
I20260925 14:57:25.636024 6818 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:25.636228 6818 heartbeater.cc:511] Master 127.20.199.62:37495 requested a full tablet report, sending...
I20260925 14:57:25.636744 6368 ts_manager.cc:194] Registered new tserver with Master: e394e0ed11e145cf89fd38d597a6c79f (127.20.199.3:38821)
I20260925 14:57:25.637095 6368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.3:41469
I20260925 14:57:25.648030 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:25.652693 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.649641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.20.199.62:37495)
0925 14:57:25.649715 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.649717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.649856 (+ 139us) server_negotiation.cc:407] Connection header received
0925 14:57:25.649931 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.649931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.649944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.650014 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:25.650202 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.650704 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.651463 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.651649 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.651714 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.652417 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:25.652418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:25.652419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:25.652423 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:25.652438 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:25.652442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:25.652500 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.652536 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.652561 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:57:25.654678 6368 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:25.656185 6368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:25.663096 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.658502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:38821 (local address 127.0.0.1:45512)
0925 14:57:25.658850 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.658891 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.659000 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.659545 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.659546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.659548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.659586 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.659588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.660806 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.660807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.661801 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.661804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.662751 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.662849 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.662864 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.662948 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.663006 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":36,"thread_start_us":82,"threads_started":1}
I20260925 14:57:25.663107 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.658613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.20.199.3:38821)
0925 14:57:25.658887 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.658894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.658962 (+ 68us) server_negotiation.cc:407] Connection header received
0925 14:57:25.659119 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.659122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.659168 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.659254 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.659649 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.660717 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.661892 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.662620 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662728 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.662820 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.662971 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.663010 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260925 14:57:25.663272 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.658709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.20.199.2:45629)
0925 14:57:25.658978 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.658983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.658994 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:25.659365 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.659368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.659384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.659472 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.659611 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.660393 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662055 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.662748 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662806 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.662885 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.663144 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.663187 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":84,"threads_started":1}
I20260925 14:57:25.663551 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.658833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.20.199.1:34677)
0925 14:57:25.659079 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.659084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.659219 (+ 135us) server_negotiation.cc:407] Connection header received
0925 14:57:25.659265 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.659268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.659281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.659360 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.659524 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.660696 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662103 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.662623 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662944 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.663045 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.663427 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.663472 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
I20260925 14:57:25.663484 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.658502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:45629 (local address 127.0.0.1:38560)
0925 14:57:25.658903 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.658916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.658999 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.659483 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.659483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.659485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.659529 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.659531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.660490 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.660491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.661960 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.661963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.662939 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.662941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.663006 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.663009 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.663062 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.663100 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":40,"thread_start_us":72,"threads_started":1}
I20260925 14:57:25.663542 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.658721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34677 (local address 127.0.0.1:57998)
0925 14:57:25.659136 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.659146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.659201 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.659374 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.659377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.659380 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.659455 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.659457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.660799 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.660800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.662027 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.662029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.663076 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.663077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.663158 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.663162 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.663285 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.663331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":68,"threads_started":1}
I20260925 14:57:25.664414 6752 tablet_service.cc:1467] Processing CreateTablet for tablet 7506474da2bc40c1a22cb29cc8c065b1 (DEFAULT_TABLE table=test-table [id=d1937881c7bf415abc72129710bb7434]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:25.664403 6621 tablet_service.cc:1467] Processing CreateTablet for tablet 7506474da2bc40c1a22cb29cc8c065b1 (DEFAULT_TABLE table=test-table [id=d1937881c7bf415abc72129710bb7434]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:25.664533 6490 tablet_service.cc:1467] Processing CreateTablet for tablet 7506474da2bc40c1a22cb29cc8c065b1 (DEFAULT_TABLE table=test-table [id=d1937881c7bf415abc72129710bb7434]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:25.664769 6621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7506474da2bc40c1a22cb29cc8c065b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:25.664769 6752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7506474da2bc40c1a22cb29cc8c065b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:25.664932 6490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7506474da2bc40c1a22cb29cc8c065b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:25.667936 6832 tablet_bootstrap.cc:492] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: Bootstrap starting.
I20260925 14:57:25.668334 6833 tablet_bootstrap.cc:492] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Bootstrap starting.
I20260925 14:57:25.668874 6832 tablet_bootstrap.cc:654] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:25.668998 6833 tablet_bootstrap.cc:654] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:25.669267 6832 log.cc:824] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:25.669339 6833 log.cc:824] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:25.669580 6834 tablet_bootstrap.cc:492] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Bootstrap starting.
I20260925 14:57:25.670241 6834 tablet_bootstrap.cc:654] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:25.670409 6832 tablet_bootstrap.cc:492] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: No bootstrap required, opened a new log
I20260925 14:57:25.670486 6833 tablet_bootstrap.cc:492] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: No bootstrap required, opened a new log
I20260925 14:57:25.670490 6832 ts_tablet_manager.cc:1397] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:57:25.670562 6834 log.cc:824] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:25.670583 6833 ts_tablet_manager.cc:1397] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:25.671419 6834 tablet_bootstrap.cc:492] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: No bootstrap required, opened a new log
I20260925 14:57:25.671495 6834 ts_tablet_manager.cc:1397] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:25.672706 6832 raft_consensus.cc:348] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.672900 6832 raft_consensus.cc:374] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:25.672966 6832 raft_consensus.cc:729] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b809a02d6b443818b374e2d33baee66, State: Initialized, Role: FOLLOWER
I20260925 14:57:25.673101 6832 consensus_queue.cc:260] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.673467 6556 heartbeater.cc:503] Master 127.20.199.62:37495 was elected leader, sending a full tablet report...
I20260925 14:57:25.673460 6833 raft_consensus.cc:348] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.673687 6833 raft_consensus.cc:374] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:25.673451 6832 ts_tablet_manager.cc:1428] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:25.673766 6833 raft_consensus.cc:729] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e394e0ed11e145cf89fd38d597a6c79f, State: Initialized, Role: FOLLOWER
I20260925 14:57:25.673924 6833 consensus_queue.cc:260] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.674171 6833 ts_tablet_manager.cc:1428] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:57:25.674196 6834 raft_consensus.cc:348] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.674340 6818 heartbeater.cc:503] Master 127.20.199.62:37495 was elected leader, sending a full tablet report...
I20260925 14:57:25.674343 6834 raft_consensus.cc:374] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:25.674413 6834 raft_consensus.cc:729] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5219a80eab4fd19df6806b26b4c39e, State: Initialized, Role: FOLLOWER
I20260925 14:57:25.674522 6834 consensus_queue.cc:260] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.674784 6834 ts_tablet_manager.cc:1428] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 14:57:25.674925 6687 heartbeater.cc:503] Master 127.20.199.62:37495 was elected leader, sending a full tablet report...
I20260925 14:57:25.726420 6839 raft_consensus.cc:482] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:25.726539 6839 raft_consensus.cc:504] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.726869 6839 leader_election.cc:288] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a5219a80eab4fd19df6806b26b4c39e (127.20.199.2:45629), 3b809a02d6b443818b374e2d33baee66 (127.20.199.1:34677)
I20260925 14:57:25.729857 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.727137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:41415 (local address 127.20.199.2:45629)
0925 14:57:25.727182 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.727184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.727326 (+ 142us) server_negotiation.cc:407] Connection header received
0925 14:57:25.727391 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.727392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.727407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.727471 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.727631 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.728085 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.729080 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.729492 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.729598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.729607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.729745 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.729783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:57:25.729900 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.727129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:49067 (local address 127.20.199.1:34677)
0925 14:57:25.727195 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.727198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.727211 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:57:25.727255 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.727256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.727277 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.727360 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:25.727542 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.728081 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.729087 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.729510 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.729612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.729621 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.729783 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.729822 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:57:25.729913 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.727048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34677 (local address 127.20.199.3:49067)
0925 14:57:25.727138 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.727150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.727200 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.727383 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.727385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.727387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.727463 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.727464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.728176 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.728177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.728969 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.728972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.729610 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.729611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.729686 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.729689 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.729768 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.729811 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":34}
I20260925 14:57:25.729916 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.727082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:45629 (local address 127.20.199.3:41415)
0925 14:57:25.727251 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.727260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:25.727335 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:25.727483 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:25.727484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:25.727486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:25.727559 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.727560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.728167 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.728168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.728983 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:25.728985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.729598 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.729599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:25.729673 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.729676 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:25.729741 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:57:25.729780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":13,"thread_start_us":120,"threads_started":1}
I20260925 14:57:25.730305 6510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7506474da2bc40c1a22cb29cc8c065b1" candidate_uuid: "e394e0ed11e145cf89fd38d597a6c79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b809a02d6b443818b374e2d33baee66" is_pre_election: true
I20260925 14:57:25.730347 6641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7506474da2bc40c1a22cb29cc8c065b1" candidate_uuid: "e394e0ed11e145cf89fd38d597a6c79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5219a80eab4fd19df6806b26b4c39e" is_pre_election: true
I20260925 14:57:25.730470 6510 raft_consensus.cc:2454] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e394e0ed11e145cf89fd38d597a6c79f in term 0.
I20260925 14:57:25.730501 6641 raft_consensus.cc:2454] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e394e0ed11e145cf89fd38d597a6c79f in term 0.
I20260925 14:57:25.730721 6707 leader_election.cc:302] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5219a80eab4fd19df6806b26b4c39e, e394e0ed11e145cf89fd38d597a6c79f; no voters:
I20260925 14:57:25.730865 6839 raft_consensus.cc:2781] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:25.730916 6839 raft_consensus.cc:482] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:25.730943 6839 raft_consensus.cc:3037] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.731889 6839 raft_consensus.cc:504] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.732031 6839 leader_election.cc:288] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [CANDIDATE]: Term 1 election: Requested vote from peers 6a5219a80eab4fd19df6806b26b4c39e (127.20.199.2:45629), 3b809a02d6b443818b374e2d33baee66 (127.20.199.1:34677)
I20260925 14:57:25.732195 6641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7506474da2bc40c1a22cb29cc8c065b1" candidate_uuid: "e394e0ed11e145cf89fd38d597a6c79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5219a80eab4fd19df6806b26b4c39e"
I20260925 14:57:25.732277 6641 raft_consensus.cc:3037] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.732242 6510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7506474da2bc40c1a22cb29cc8c065b1" candidate_uuid: "e394e0ed11e145cf89fd38d597a6c79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b809a02d6b443818b374e2d33baee66"
I20260925 14:57:25.732337 6510 raft_consensus.cc:3037] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:25.733055 6641 raft_consensus.cc:2454] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e394e0ed11e145cf89fd38d597a6c79f in term 1.
I20260925 14:57:25.733183 6510 raft_consensus.cc:2454] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e394e0ed11e145cf89fd38d597a6c79f in term 1.
I20260925 14:57:25.733208 6707 leader_election.cc:302] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5219a80eab4fd19df6806b26b4c39e, e394e0ed11e145cf89fd38d597a6c79f; no voters:
I20260925 14:57:25.733280 6839 raft_consensus.cc:2781] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:25.733425 6839 raft_consensus.cc:686] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 1 LEADER]: Becoming Leader. State: Replica: e394e0ed11e145cf89fd38d597a6c79f, State: Running, Role: LEADER
I20260925 14:57:25.733515 6839 consensus_queue.cc:237] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } }
I20260925 14:57:25.734396 6367 catalog_manager.cc:5495] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f reported cstate change: term changed from 0 to 1, leader changed from <none> to e394e0ed11e145cf89fd38d597a6c79f (127.20.199.3). New cstate: current_term: 1 leader_uuid: "e394e0ed11e145cf89fd38d597a6c79f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e394e0ed11e145cf89fd38d597a6c79f" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 } health_report { overall_health: UNKNOWN } } }
W20260925 14:57:25.746788 6688 tablet.cc:2369] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:25.754534 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.752145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.20.199.3:38821)
0925 14:57:25.752208 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:25.752210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:25.752218 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:25.752251 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:25.752252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:25.752267 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:25.752338 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:25.752487 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.752985 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.753805 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:25.754023 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:25.754080 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:25.754252 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:25.754347 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:25.754406 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:25.754446 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 14:57:25.754461 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:57:25.756376 6641 raft_consensus.cc:1261] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 1 FOLLOWER]: Refusing update from remote peer e394e0ed11e145cf89fd38d597a6c79f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:25.756402 6510 raft_consensus.cc:1261] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66 [term 1 FOLLOWER]: Refusing update from remote peer e394e0ed11e145cf89fd38d597a6c79f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:25.756714 6839 consensus_queue.cc:1035] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b809a02d6b443818b374e2d33baee66" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:25.756839 6842 consensus_queue.cc:1035] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a5219a80eab4fd19df6806b26b4c39e" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:25.759225 6847 mvcc.cc:205] Tried to move back new op lower bound from 7333266414615240704 to 7333266414525063168. Current Snapshot: MvccSnapshot[applied={T|T < 7333266414615240704}]
I20260925 14:57:25.761375 6367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:25.761523 6367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:25.762188 6367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0fc876ec1f54f68a38e5cc2a10ac7ba: Sending DeleteTablet for 3 replicas of tablet 7506474da2bc40c1a22cb29cc8c065b1
I20260925 14:57:25.762512 6621 tablet_service.cc:1514] Processing DeleteTablet for tablet 7506474da2bc40c1a22cb29cc8c065b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:25 UTC) from {username='slave'} at 127.0.0.1:38560
I20260925 14:57:25.762579 6752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7506474da2bc40c1a22cb29cc8c065b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:25 UTC) from {username='slave'} at 127.0.0.1:45512
I20260925 14:57:25.762647 6490 tablet_service.cc:1514] Processing DeleteTablet for tablet 7506474da2bc40c1a22cb29cc8c065b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:25 UTC) from {username='slave'} at 127.0.0.1:57998
I20260925 14:57:25.762912 6856 tablet_replica.cc:330] T 7506474da2bc40c1a22cb29cc8c065b1 P 3b809a02d6b443818b374e2d33baee66: stopping tablet replica
I20260925 14:57:25.762965 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6428
I20260925 14:57:25.763566 6854 tablet_replica.cc:330] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: stopping tablet replica
I20260925 14:57:25.763648 6854 raft_consensus.cc:2229] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:25.763722 6854 raft_consensus.cc:2258] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:25.763877 6855 tablet_replica.cc:330] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: stopping tablet replica
I20260925 14:57:25.763959 6855 raft_consensus.cc:2229] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:25.764045 6855 raft_consensus.cc:2258] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:25.764184 6854 ts_tablet_manager.cc:1905] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:25.764477 6855 ts_tablet_manager.cc:1905] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:25.765833 6854 ts_tablet_manager.cc:1918] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:25.765909 6854 log.cc:1196] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/7506474da2bc40c1a22cb29cc8c065b1
I20260925 14:57:25.766106 6855 ts_tablet_manager.cc:1918] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:25.766157 6854 ts_tablet_manager.cc:1939] T 7506474da2bc40c1a22cb29cc8c065b1 P 6a5219a80eab4fd19df6806b26b4c39e: Deleting consensus metadata
I20260925 14:57:25.766165 6855 log.cc:1196] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/7506474da2bc40c1a22cb29cc8c065b1
I20260925 14:57:25.766403 6855 ts_tablet_manager.cc:1939] T 7506474da2bc40c1a22cb29cc8c065b1 P e394e0ed11e145cf89fd38d597a6c79f: Deleting consensus metadata
I20260925 14:57:25.766556 6354 catalog_manager.cc:4841] TS 6a5219a80eab4fd19df6806b26b4c39e (127.20.199.2:45629): tablet 7506474da2bc40c1a22cb29cc8c065b1 (table test-table [id=d1937881c7bf415abc72129710bb7434]) successfully deleted
I20260925 14:57:25.766983 6353 catalog_manager.cc:4841] TS e394e0ed11e145cf89fd38d597a6c79f (127.20.199.3:38821): tablet 7506474da2bc40c1a22cb29cc8c065b1 (table test-table [id=d1937881c7bf415abc72129710bb7434]) successfully deleted
W20260925 14:57:25.771515 6353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:34677 (error 108)
I20260925 14:57:25.771695 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6559
I20260925 14:57:25.772431 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:25.772290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34677 (local address 127.0.0.1:58004)
0925 14:57:25.772337 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:25.772374 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 14:57:25.772540 6353 catalog_manager.cc:4584] TS 3b809a02d6b443818b374e2d33baee66 (127.20.199.1:34677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7506474da2bc40c1a22cb29cc8c065b1: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34677: connect: Connection refused (error 111)
I20260925 14:57:25.776831 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6690
I20260925 14:57:25.780829 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6337
2026-09-25T14:57:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (715 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:25 dist-test-slave-8qt6 krb5kdc[6860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:25 dist-test-slave-8qt6 krb5kdc[6860](info): set up 2 sockets
Sep 25 14:57:25 dist-test-slave-8qt6 krb5kdc[6860](info): commencing operation
krb5kdc: starting...
W20260925 14:57:26.817212 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:26 dist-test-slave-8qt6 krb5kdc[6860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:26.936257 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37577
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:37577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:27.016031 6873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:27.016285 6873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:27.016326 6873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:27.017483 6873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:27.017552 6873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:27.017577 6873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:27.017601 6873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:27.017628 6873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:27.019094 6873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:37577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:27.019369 6873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:27.019595 6873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:27.024366 6878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.024343 6881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.024487 6879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.024746 6873 server_base.cc:1034] running on GCE node
I20260925 14:57:27.024983 6873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:27.025312 6873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:27.026476 6873 hybrid_clock.cc:648] HybridClock initialized: now 1790348247026476 us; error 58 us; skew 500 ppm
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:27.028939 6873 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:27.029392 6873 webserver.cc:460] Webserver started at http://127.20.199.62:40549/ using document root <none> and password file <none>
I20260925 14:57:27.029664 6873 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:27.029708 6873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:27.029798 6873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:27.030790 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "f4e55ea7b7ef425c98c844fd90b87b3e"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "15c043e55e10fea2ca4672dd9be64d12"
server_key_iv: "1a1019e92400ab3395f8b043828db926"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.031095 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "f4e55ea7b7ef425c98c844fd90b87b3e"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "15c043e55e10fea2ca4672dd9be64d12"
server_key_iv: "1a1019e92400ab3395f8b043828db926"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.032341 6873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:27.033023 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.033200 6873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:27.033293 6873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "f4e55ea7b7ef425c98c844fd90b87b3e"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "15c043e55e10fea2ca4672dd9be64d12"
server_key_iv: "1a1019e92400ab3395f8b043828db926"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.033370 6873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:27.048384 6873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:27.050740 6873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:27.050945 6873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:27.054677 6873 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:37577
I20260925 14:57:27.054731 6940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:37577 every 8 connection(s)
I20260925 14:57:27.055094 6873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:27.056052 6941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:27.058581 6941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e: Bootstrap starting.
I20260925 14:57:27.059175 6941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:27.059494 6941 log.cc:824] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:27.060221 6941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e: No bootstrap required, opened a new log
I20260925 14:57:27.060961 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6873
I20260925 14:57:27.061110 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:27.061303 21276 external_mini_cluster.cc:1428] Setting key 3fea69cf743ad488e06c58f7b1cc6738
I20260925 14:57:27.062402 6941 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37577 } }
I20260925 14:57:27.062568 6941 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:27.062633 6941 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e55ea7b7ef425c98c844fd90b87b3e, State: Initialized, Role: FOLLOWER
I20260925 14:57:27.062814 6941 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37577 } }
I20260925 14:57:27.062911 6941 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:27.062956 6941 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:27.063007 6941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:27.063875 6941 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37577 } }
I20260925 14:57:27.063995 6941 leader_election.cc:302] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e55ea7b7ef425c98c844fd90b87b3e; no voters:
I20260925 14:57:27.064195 6941 leader_election.cc:288] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:27.064217 6946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:27.064327 6946 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [term 1 LEADER]: Becoming Leader. State: Replica: f4e55ea7b7ef425c98c844fd90b87b3e, State: Running, Role: LEADER
I20260925 14:57:27.064476 6941 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:27.064477 6946 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37577 } }
I20260925 14:57:27.064952 6947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e55ea7b7ef425c98c844fd90b87b3e. Latest consensus state: current_term: 1 leader_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37577 } } }
I20260925 14:57:27.065047 6947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:27.065279 6948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e55ea7b7ef425c98c844fd90b87b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37577 } } }
I20260925 14:57:27.065608 6953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:27.066234 6953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:27.067206 6948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:27.067759 6953 catalog_manager.cc:1347] Generated new cluster ID: 67d96412ef4b4e958791ff32e9a1dfab
I20260925 14:57:27.067809 6953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:27.072528 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.061944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.20.199.62:37577)
0925 14:57:27.062137 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.062145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.062171 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:57:27.062356 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.062362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.062406 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.062580 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.063384 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.064421 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.065526 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.066157 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.066259 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.068869 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.068873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.068873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:27.068885 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.070613 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.071000 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.071003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.071003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.071125 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.071341 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.071342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.071342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.071582 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.072194 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.072273 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.072381 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260925 14:57:27.113073 6953 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:27.113762 6953 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:27.122596 6953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e: Generated new TSK 0
I20260925 14:57:27.122852 6953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:57:27.157166 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37577
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:27.239246 6967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:27.239445 6967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:27.239471 6967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:27.240624 6967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:27.240669 6967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:27.240721 6967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:27.242282 6967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:27.242527 6967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:27.242769 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:27.247768 6972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.247784 6973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.247829 6967 server_base.cc:1034] running on GCE node
W20260925 14:57:27.247778 6975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.248173 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:27.248456 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:27.249608 6967 hybrid_clock.cc:648] HybridClock initialized: now 1790348247249597 us; error 29 us; skew 500 ppm
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:27.252110 6967 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:57:27.252523 6967 webserver.cc:460] Webserver started at http://127.20.199.1:40575/ using document root <none> and password file <none>
I20260925 14:57:27.252745 6967 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:27.252815 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:27.252967 6967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:27.254089 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "90acbbc922eb465e8c13a36e08c2a2fb"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "8e0df3dd80cab73440e6f4878bcfb08a"
server_key_iv: "39778dd732a209ae8b52a7233acc64e9"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.254455 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "90acbbc922eb465e8c13a36e08c2a2fb"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "8e0df3dd80cab73440e6f4878bcfb08a"
server_key_iv: "39778dd732a209ae8b52a7233acc64e9"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.255769 6967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:27.256496 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.256675 6967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:27.256776 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "90acbbc922eb465e8c13a36e08c2a2fb"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "8e0df3dd80cab73440e6f4878bcfb08a"
server_key_iv: "39778dd732a209ae8b52a7233acc64e9"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.256876 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:27.268314 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:27.270177 6967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:27.270340 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:27.270567 6967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:27.270875 6967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:27.270927 6967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.270963 6967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:27.270991 6967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.276896 6967 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:34693
I20260925 14:57:27.276938 7095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:34693 every 8 connection(s)
I20260925 14:57:27.277277 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:27.282015 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 6967
I20260925 14:57:27.282140 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:27.282388 21276 external_mini_cluster.cc:1428] Setting key a427d9f7aae09d1e6accdeada1e59aa0
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:27.287057 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.278123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58245 (local address 127.20.199.62:37577)
0925 14:57:27.278168 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.278170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.278387 (+ 217us) server_negotiation.cc:407] Connection header received
0925 14:57:27.278920 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.278921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.278937 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.279002 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.279990 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.280497 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.281519 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.281735 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.281850 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.284031 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.284032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.284033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:27.284044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.285544 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.286050 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.286051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.286051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.286108 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.286392 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.286393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.286393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.286495 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.286547 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.286882 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.286954 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:27.287348 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.278097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37577 (local address 127.20.199.1:58245)
0925 14:57:27.278267 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.278301 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.278810 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.279072 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.279076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.279675 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:27.279871 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.279875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.280598 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.280600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.281426 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.281427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.281856 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.281858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.281989 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.282576 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:27.282584 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:27.283905 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:27.285640 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:27.285648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:27.285650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:27.285969 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:27.286179 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:27.286180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:27.286181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:27.286328 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:27.286564 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:27.286570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:27.286737 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.286860 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.287053 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260925 14:57:27.287875 7096 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37577
I20260925 14:57:27.287994 7096 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:27.288206 7096 heartbeater.cc:511] Master 127.20.199.62:37577 requested a full tablet report, sending...
I20260925 14:57:27.288733 6905 ts_manager.cc:194] Registered new tserver with Master: 90acbbc922eb465e8c13a36e08c2a2fb (127.20.199.1:34693)
I20260925 14:57:27.289333 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:58245
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:57:27.308674 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37577
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:27.391105 7101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:27.391312 7101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:27.391336 7101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:27.392506 7101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:27.392550 7101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:27.392576 7101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:27.394112 7101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:27.394353 7101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:27.394630 7101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:27.399864 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.399919 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.399873 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.399969 7101 server_base.cc:1034] running on GCE node
I20260925 14:57:27.400291 7101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:27.400547 7101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:27.401706 7101 hybrid_clock.cc:648] HybridClock initialized: now 1790348247401684 us; error 38 us; skew 500 ppm
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:27.404146 7101 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:57:27.404630 7101 webserver.cc:460] Webserver started at http://127.20.199.2:40727/ using document root <none> and password file <none>
I20260925 14:57:27.404870 7101 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:27.404915 7101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:27.405081 7101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:27.406185 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "dcd7305a3e084d249878bb2b6efc786d"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "20470f4b03a579d2c196ea0748bf72f3"
server_key_iv: "e2936b0bccfbad5ec4c72ee0d64da8ef"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.406541 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "dcd7305a3e084d249878bb2b6efc786d"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "20470f4b03a579d2c196ea0748bf72f3"
server_key_iv: "e2936b0bccfbad5ec4c72ee0d64da8ef"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.407871 7101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:57:27.408650 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.408850 7101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:27.408948 7101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "dcd7305a3e084d249878bb2b6efc786d"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "20470f4b03a579d2c196ea0748bf72f3"
server_key_iv: "e2936b0bccfbad5ec4c72ee0d64da8ef"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.409030 7101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:27.423712 7101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:27.425786 7101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:27.425953 7101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:27.426184 7101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:27.426503 7101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:27.426563 7101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.426607 7101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:27.426635 7101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.432643 7101 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:41347
I20260925 14:57:27.432683 7229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:41347 every 8 connection(s)
I20260925 14:57:27.432996 7101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:27.433504 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7101
I20260925 14:57:27.433611 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:27.433833 21276 external_mini_cluster.cc:1428] Setting key 0a6d2561298f53f8ebbcc02d629558d9
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
I20260925 14:57:27.444402 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.433710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:39983 (local address 127.20.199.62:37577)
0925 14:57:27.433789 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.433792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.434169 (+ 377us) server_negotiation.cc:407] Connection header received
0925 14:57:27.434809 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.434810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.434819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.434857 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.436205 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.437083 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.438706 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.438934 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.439012 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.441580 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.441582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.441582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:27.441591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.442982 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.443389 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.443390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.443391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.443452 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.443723 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.443724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.443724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.443845 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.443951 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.444221 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.444309 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:57:27.444613 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.433701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37577 (local address 127.20.199.2:39983)
0925 14:57:27.433928 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.434086 (+ 158us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.434704 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.434941 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.434945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.435778 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:27.436093 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.436101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.437183 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.437184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.438589 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.438593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.439044 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.439045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.439208 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.439759 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:27.439770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:27.441483 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:27.443070 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:27.443077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:27.443080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:27.443308 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:27.443523 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:27.443524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:27.443524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:27.443648 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:27.443921 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:27.443927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:27.444087 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.444201 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.444367 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
I20260925 14:57:27.445151 7230 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37577
I20260925 14:57:27.445253 7230 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:27.445446 7230 heartbeater.cc:511] Master 127.20.199.62:37577 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
I20260925 14:57:27.445852 6905 ts_manager.cc:194] Registered new tserver with Master: dcd7305a3e084d249878bb2b6efc786d (127.20.199.2:41347)
I20260925 14:57:27.446223 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:39983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:57:27.460427 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37577
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:27.541494 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:27.541706 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:27.541730 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:27.542833 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:27.542876 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:27.542902 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:27.544268 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:27.544453 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:27.544625 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:27.549708 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.549715 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.549708 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.549731 7235 server_base.cc:1034] running on GCE node
I20260925 14:57:27.550263 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:27.550565 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:27.551743 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790348247551740 us; error 37 us; skew 500 ppm
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:27.554376 7235 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:57:27.554839 7235 webserver.cc:460] Webserver started at http://127.20.199.3:32941/ using document root <none> and password file <none>
I20260925 14:57:27.555092 7235 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:27.555163 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:27.555316 7235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:27.556371 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "9c89f43f3f6944f2aa05aa9706568d43"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "c55f318c9192fa50b7d233c9e72872d3"
server_key_iv: "d7e57f2d84d1d78e4ef629851b77e3d8"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.556747 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "9c89f43f3f6944f2aa05aa9706568d43"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "c55f318c9192fa50b7d233c9e72872d3"
server_key_iv: "d7e57f2d84d1d78e4ef629851b77e3d8"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.558135 7235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:57:27.558843 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.559031 7235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:27.559135 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "9c89f43f3f6944f2aa05aa9706568d43"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "c55f318c9192fa50b7d233c9e72872d3"
server_key_iv: "d7e57f2d84d1d78e4ef629851b77e3d8"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.559208 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:27.569286 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:27.571630 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:27.571794 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:27.572023 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:27.572327 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:27.572386 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.572440 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:27.572464 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.578266 7235 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:34529
I20260925 14:57:27.578308 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:34529 every 8 connection(s)
I20260925 14:57:27.578651 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:57:27 dist-test-slave-8qt6 krb5kdc[6860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:27.585178 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7235
I20260925 14:57:27.585292 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:27.585579 21276 external_mini_cluster.cc:1428] Setting key ef751ba6bbb8d07a9df819e3cd0258f9
I20260925 14:57:27.588049 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.579370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:57683 (local address 127.20.199.62:37577)
0925 14:57:27.579437 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.579439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.579655 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:57:27.580176 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.580177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.580189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.580248 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.581181 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.581740 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.582734 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.582932 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.583044 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.585093 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.585094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.585095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:27.585102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.586638 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.587067 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.587068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.587069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.587134 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.587389 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.587390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.587391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.587516 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.587587 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.587868 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.587951 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:57:27.588236 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.579342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37577 (local address 127.20.199.3:57683)
0925 14:57:27.579538 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.579573 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.580070 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.580318 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.580322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.580886 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:27.581078 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.581083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.581831 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.581832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.582634 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.582636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.583054 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.583056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.583183 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.583633 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:27.583641 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:27.584938 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:27.586744 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:27.586750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:27.586752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:27.586973 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:27.587202 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:27.587203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:27.587204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:27.587319 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:27.587590 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:27.587597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:27.587734 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.587844 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.588004 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260925 14:57:27.588722 7364 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37577
I20260925 14:57:27.588800 7364 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:27.589011 7364 heartbeater.cc:511] Master 127.20.199.62:37577 requested a full tablet report, sending...
I20260925 14:57:27.589468 6905 ts_manager.cc:194] Registered new tserver with Master: 9c89f43f3f6944f2aa05aa9706568d43 (127.20.199.3:34529)
I20260925 14:57:27.589862 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:57683
I20260925 14:57:27.597256 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:27.604863 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.598681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.20.199.62:37577)
0925 14:57:27.598747 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.598749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.598859 (+ 110us) server_negotiation.cc:407] Connection header received
0925 14:57:27.598906 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.598907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.598916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.598969 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.599475 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.599941 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.600705 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.600890 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.600944 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.602303 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.602304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.602304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:27.602311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.603737 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.604109 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.604110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.604110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.604164 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:27.604364 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:27.604364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:27.604365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:27.604495 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.604572 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.604689 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.604738 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:57:27.606413 6905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:27.607975 6905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:27.614236 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.610141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34529 (local address 127.0.0.1:58342)
0925 14:57:27.610446 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.610484 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.610554 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.611017 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.611021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.611024 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:27.611102 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.611104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.612167 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.612168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.613238 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.613240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.614021 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.614022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.614079 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.614090 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.614145 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.614181 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":82,"threads_started":1}
I20260925 14:57:27.614269 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.610319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.20.199.3:34529)
0925 14:57:27.610795 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.610799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.610808 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:27.610843 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.610846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.610861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.610928 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:27.611186 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.612069 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.613320 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.613938 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.614015 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.614073 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.614158 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.614191 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":73,"threads_started":1}
I20260925 14:57:27.614892 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.610142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34693 (local address 127.0.0.1:48034)
0925 14:57:27.610660 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.610669 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.610718 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.611057 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.611058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.611060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:27.611108 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.611110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.612442 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.612443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.613740 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.613744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.614662 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.614664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.614731 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.614735 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.614782 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.614824 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":83,"thread_start_us":80,"threads_started":1}
I20260925 14:57:27.615000 7298 tablet_service.cc:1467] Processing CreateTablet for tablet ad4ca427fdce421988b6cdf2cdb6b433 (DEFAULT_TABLE table=test-table [id=3637eeea391a4d3d8ef57a216554f52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:27.615388 7298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4ca427fdce421988b6cdf2cdb6b433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:27.615433 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.610245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.20.199.1:34693)
0925 14:57:27.610678 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.610682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.610741 (+ 59us) server_negotiation.cc:407] Connection header received
0925 14:57:27.610799 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.610801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.610820 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.610918 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:27.611204 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.612330 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.613833 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.614582 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.614650 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.614702 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.615008 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.615057 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":94,"threads_started":1}
I20260925 14:57:27.615900 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.610358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:41347 (local address 127.0.0.1:49996)
0925 14:57:27.610792 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.610798 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.610851 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.611090 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.611091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.611092 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:27.611120 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.611121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.612439 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.612439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.613742 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.613743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.615681 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.615682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.615743 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.615747 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.615795 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.615836 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":56,"thread_start_us":53,"threads_started":1}
I20260925 14:57:27.616204 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.610492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.20.199.2:41347)
0925 14:57:27.610680 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.610686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.610851 (+ 165us) server_negotiation.cc:407] Connection header received
0925 14:57:27.610918 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.610920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.610936 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.611007 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:27.611194 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.612316 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.614741 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.615608 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.615680 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.615732 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.616078 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.616127 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260925 14:57:27.616204 7030 tablet_service.cc:1467] Processing CreateTablet for tablet ad4ca427fdce421988b6cdf2cdb6b433 (DEFAULT_TABLE table=test-table [id=3637eeea391a4d3d8ef57a216554f52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:27.616531 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4ca427fdce421988b6cdf2cdb6b433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:27.618211 7164 tablet_service.cc:1467] Processing CreateTablet for tablet ad4ca427fdce421988b6cdf2cdb6b433 (DEFAULT_TABLE table=test-table [id=3637eeea391a4d3d8ef57a216554f52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:27.618543 7164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4ca427fdce421988b6cdf2cdb6b433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:27.619470 7379 tablet_bootstrap.cc:492] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb: Bootstrap starting.
I20260925 14:57:27.620143 7379 tablet_bootstrap.cc:654] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:27.620730 7379 log.cc:824] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:27.621605 7379 tablet_bootstrap.cc:492] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb: No bootstrap required, opened a new log
I20260925 14:57:27.621726 7378 tablet_bootstrap.cc:492] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Bootstrap starting.
I20260925 14:57:27.621765 7379 ts_tablet_manager.cc:1397] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:27.622400 7378 tablet_bootstrap.cc:654] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:27.622903 7378 log.cc:824] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:27.623351 7379 raft_consensus.cc:348] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.623523 7379 raft_consensus.cc:374] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:27.623591 7379 raft_consensus.cc:729] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90acbbc922eb465e8c13a36e08c2a2fb, State: Initialized, Role: FOLLOWER
I20260925 14:57:27.623682 7378 tablet_bootstrap.cc:492] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: No bootstrap required, opened a new log
I20260925 14:57:27.623754 7378 ts_tablet_manager.cc:1397] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:27.623744 7379 consensus_queue.cc:260] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.624048 7379 ts_tablet_manager.cc:1428] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:27.624068 7096 heartbeater.cc:503] Master 127.20.199.62:37577 was elected leader, sending a full tablet report...
I20260925 14:57:27.624258 7381 tablet_bootstrap.cc:492] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: Bootstrap starting.
I20260925 14:57:27.624785 7381 tablet_bootstrap.cc:654] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:27.625300 7381 log.cc:824] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:27.625888 7378 raft_consensus.cc:348] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.626098 7378 raft_consensus.cc:374] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:27.626139 7381 tablet_bootstrap.cc:492] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: No bootstrap required, opened a new log
I20260925 14:57:27.626152 7378 raft_consensus.cc:729] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c89f43f3f6944f2aa05aa9706568d43, State: Initialized, Role: FOLLOWER
I20260925 14:57:27.626196 7381 ts_tablet_manager.cc:1397] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:27.626271 7378 consensus_queue.cc:260] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.626549 7378 ts_tablet_manager.cc:1428] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:27.626573 7364 heartbeater.cc:503] Master 127.20.199.62:37577 was elected leader, sending a full tablet report...
I20260925 14:57:27.628057 7381 raft_consensus.cc:348] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.628191 7381 raft_consensus.cc:374] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:27.628212 7381 raft_consensus.cc:729] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd7305a3e084d249878bb2b6efc786d, State: Initialized, Role: FOLLOWER
I20260925 14:57:27.628321 7381 consensus_queue.cc:260] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.628580 7381 ts_tablet_manager.cc:1428] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:57:27.628612 7230 heartbeater.cc:503] Master 127.20.199.62:37577 was elected leader, sending a full tablet report...
I20260925 14:57:27.634074 7383 raft_consensus.cc:482] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:27.634169 7383 raft_consensus.cc:504] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.634475 7383 leader_election.cc:288] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd7305a3e084d249878bb2b6efc786d (127.20.199.2:41347), 9c89f43f3f6944f2aa05aa9706568d43 (127.20.199.3:34529)
I20260925 14:57:27.637615 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.634618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:41347 (local address 127.20.199.1:47361)
0925 14:57:27.634662 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.634674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.634719 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.635057 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.635058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.635059 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:27.635125 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.635126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.635840 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.635841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.636709 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.636712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.637324 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.637325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.637423 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.637427 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.637480 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.637531 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:57:27.637631 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.634820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:47361 (local address 127.20.199.2:41347)
0925 14:57:27.634922 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.634924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.634934 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:27.634958 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.634959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.634969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.635038 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:27.635225 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.635743 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.636796 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.637243 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.637327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.637335 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.637498 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.637542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:57:27.637750 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.634684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:55651 (local address 127.20.199.3:34529)
0925 14:57:27.634734 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.634736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.634891 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:57:27.634947 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.634948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.634956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.634999 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:27.635178 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.635734 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.636828 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.637211 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.637313 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.637323 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.637643 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.637689 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:57:27.637967 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.634631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:34529 (local address 127.20.199.1:55651)
0925 14:57:27.634788 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.634798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.634884 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.635021 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.635022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.635024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:27.635105 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.635112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.635831 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.635832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.636743 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.636745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.637483 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.637484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.637554 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.637558 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.637846 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.637895 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260925 14:57:27.637938 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ca427fdce421988b6cdf2cdb6b433" candidate_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd7305a3e084d249878bb2b6efc786d" is_pre_election: true
I20260925 14:57:27.638085 7184 raft_consensus.cc:2454] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90acbbc922eb465e8c13a36e08c2a2fb in term 0.
I20260925 14:57:27.638329 6986 leader_election.cc:302] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90acbbc922eb465e8c13a36e08c2a2fb, dcd7305a3e084d249878bb2b6efc786d; no voters:
I20260925 14:57:27.638356 7318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ca427fdce421988b6cdf2cdb6b433" candidate_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c89f43f3f6944f2aa05aa9706568d43" is_pre_election: true
I20260925 14:57:27.638458 7383 raft_consensus.cc:2781] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:27.638516 7383 raft_consensus.cc:482] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:27.638525 7318 raft_consensus.cc:2454] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90acbbc922eb465e8c13a36e08c2a2fb in term 0.
I20260925 14:57:27.638548 7383 raft_consensus.cc:3037] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:27.639248 7383 raft_consensus.cc:504] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.639405 7383 leader_election.cc:288] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [CANDIDATE]: Term 1 election: Requested vote from peers dcd7305a3e084d249878bb2b6efc786d (127.20.199.2:41347), 9c89f43f3f6944f2aa05aa9706568d43 (127.20.199.3:34529)
I20260925 14:57:27.639526 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ca427fdce421988b6cdf2cdb6b433" candidate_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd7305a3e084d249878bb2b6efc786d"
I20260925 14:57:27.639547 7318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ca427fdce421988b6cdf2cdb6b433" candidate_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c89f43f3f6944f2aa05aa9706568d43"
I20260925 14:57:27.639621 7184 raft_consensus.cc:3037] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:27.639619 7318 raft_consensus.cc:3037] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:27.640298 7184 raft_consensus.cc:2454] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90acbbc922eb465e8c13a36e08c2a2fb in term 1.
I20260925 14:57:27.640350 7318 raft_consensus.cc:2454] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90acbbc922eb465e8c13a36e08c2a2fb in term 1.
I20260925 14:57:27.640501 6986 leader_election.cc:302] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90acbbc922eb465e8c13a36e08c2a2fb, dcd7305a3e084d249878bb2b6efc786d; no voters:
I20260925 14:57:27.640585 7383 raft_consensus.cc:2781] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:27.640722 7383 raft_consensus.cc:686] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [term 1 LEADER]: Becoming Leader. State: Replica: 90acbbc922eb465e8c13a36e08c2a2fb, State: Running, Role: LEADER
I20260925 14:57:27.640826 7383 consensus_queue.cc:237] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } }
I20260925 14:57:27.641628 6905 catalog_manager.cc:5495] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 90acbbc922eb465e8c13a36e08c2a2fb (127.20.199.1). New cstate: current_term: 1 leader_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90acbbc922eb465e8c13a36e08c2a2fb" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 34693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 } health_report { overall_health: UNKNOWN } } }
I20260925 14:57:27.649263 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.647075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.20.199.1:34693)
0925 14:57:27.647124 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.647126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.647133 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:27.647189 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.647189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.647205 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.647268 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:27.647357 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.647799 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.648515 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.648694 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.648748 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.648917 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:27.649008 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:27.649078 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.649128 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.649136 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:57:27.651011 7184 raft_consensus.cc:1261] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 1 FOLLOWER]: Refusing update from remote peer 90acbbc922eb465e8c13a36e08c2a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:27.651044 7318 raft_consensus.cc:1261] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 1 FOLLOWER]: Refusing update from remote peer 90acbbc922eb465e8c13a36e08c2a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:27.651358 7388 consensus_queue.cc:1035] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd7305a3e084d249878bb2b6efc786d" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:27.651486 7383 consensus_queue.cc:1035] T ad4ca427fdce421988b6cdf2cdb6b433 P 90acbbc922eb465e8c13a36e08c2a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c89f43f3f6944f2aa05aa9706568d43" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:27.653517 7391 mvcc.cc:205] Tried to move back new op lower bound from 7333266422375936000 to 7333266422337490944. Current Snapshot: MvccSnapshot[applied={T|T < 7333266422375936000}]
I20260925 14:57:27.654171 7392 mvcc.cc:205] Tried to move back new op lower bound from 7333266422375936000 to 7333266422337490944. Current Snapshot: MvccSnapshot[applied={T|T < 7333266422375936000}]
I20260925 14:57:27.654202 7393 mvcc.cc:205] Tried to move back new op lower bound from 7333266422375936000 to 7333266422337490944. Current Snapshot: MvccSnapshot[applied={T|T < 7333266422375936000}]
I20260925 14:57:27.655843 6903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:27.655953 6903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:27.656695 6903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4e55ea7b7ef425c98c844fd90b87b3e: Sending DeleteTablet for 3 replicas of tablet ad4ca427fdce421988b6cdf2cdb6b433
I20260925 14:57:27.657069 7298 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4ca427fdce421988b6cdf2cdb6b433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:27 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:58342
I20260925 14:57:27.657117 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4ca427fdce421988b6cdf2cdb6b433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:27 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:48034
I20260925 14:57:27.657181 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6967
I20260925 14:57:27.657310 7164 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4ca427fdce421988b6cdf2cdb6b433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:27 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:49996
I20260925 14:57:27.661520 7403 tablet_replica.cc:330] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: stopping tablet replica
I20260925 14:57:27.661651 7403 raft_consensus.cc:2229] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:27.661687 7401 tablet_replica.cc:330] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: stopping tablet replica
I20260925 14:57:27.661768 7401 raft_consensus.cc:2229] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:27.661773 7403 raft_consensus.cc:2258] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:27.661862 7401 raft_consensus.cc:2258] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:27.662331 7401 ts_tablet_manager.cc:1905] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:27.662350 7403 ts_tablet_manager.cc:1905] T ad4ca427fdce421988b6cdf2cdb6b433 P dcd7305a3e084d249878bb2b6efc786d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:57:27.663161 6892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:34693 (error 108)
I20260925 14:57:27.663389 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7101
I20260925 14:57:27.664450 7401 ts_tablet_manager.cc:1918] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:27.664522 7401 log.cc:1196] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/ad4ca427fdce421988b6cdf2cdb6b433
I20260925 14:57:27.664875 7401 ts_tablet_manager.cc:1939] T ad4ca427fdce421988b6cdf2cdb6b433 P 9c89f43f3f6944f2aa05aa9706568d43: Deleting consensus metadata
I20260925 14:57:27.665285 6889 catalog_manager.cc:4841] TS 9c89f43f3f6944f2aa05aa9706568d43 (127.20.199.3:34529): tablet ad4ca427fdce421988b6cdf2cdb6b433 (table test-table [id=3637eeea391a4d3d8ef57a216554f52d]) successfully deleted
I20260925 14:57:27.665735 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.665627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:34693 (local address 127.0.0.1:48048)
0925 14:57:27.665663 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.665697 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 14:57:27.665879 6892 catalog_manager.cc:4584] TS 90acbbc922eb465e8c13a36e08c2a2fb (127.20.199.1:34693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad4ca427fdce421988b6cdf2cdb6b433: Network error: Client connection negotiation failed: client connection to 127.20.199.1:34693: connect: Connection refused (error 111)
I20260925 14:57:27.667892 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7235
I20260925 14:57:27.667951 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.667834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:41347 (local address 127.0.0.1:50004)
0925 14:57:27.667895 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.667921 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.2:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:57:27.671644 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 6873
2026-09-25T14:57:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1891 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T14:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:27Z Disabled control of system clock
I20260925 14:57:27.699487 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:33171
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:33171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:27.783375 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:27.783599 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:27.783620 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:27.784750 7408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:27.784788 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:27.784801 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:27.784813 7408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:27.784825 7408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:27.786170 7408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:33171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:33171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:27.786366 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:27.786613 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:27.791424 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.791397 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.791394 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.791640 7408 server_base.cc:1034] running on GCE node
I20260925 14:57:27.791909 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:27.792188 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:27.793367 7408 hybrid_clock.cc:648] HybridClock initialized: now 1790348247793344 us; error 36 us; skew 500 ppm
I20260925 14:57:27.794770 7408 webserver.cc:460] Webserver started at http://127.20.199.62:34013/ using document root <none> and password file <none>
I20260925 14:57:27.795046 7408 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:27.795104 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:27.795259 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:27.796376 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "e005ca758f774bf2add1f97406a15853"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "916cde912599e0fc9d746abafacfa69d"
server_key_iv: "6567893a3859c0eb49ae132e0202ee19"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.796744 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "e005ca758f774bf2add1f97406a15853"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "916cde912599e0fc9d746abafacfa69d"
server_key_iv: "6567893a3859c0eb49ae132e0202ee19"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.798231 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:27.799027 7422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.799214 7408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:27.799304 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "e005ca758f774bf2add1f97406a15853"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "916cde912599e0fc9d746abafacfa69d"
server_key_iv: "6567893a3859c0eb49ae132e0202ee19"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.799384 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:27.812302 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:27.812606 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:27.812759 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:27.816908 7408 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:33171
I20260925 14:57:27.816958 7474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:33171 every 8 connection(s)
I20260925 14:57:27.817348 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:27.818356 7475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:27.820562 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: Bootstrap starting.
I20260925 14:57:27.821105 7475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:27.821352 7475 log.cc:824] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:27.822031 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: No bootstrap required, opened a new log
I20260925 14:57:27.823563 7475 raft_consensus.cc:348] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005ca758f774bf2add1f97406a15853" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33171 } }
I20260925 14:57:27.823704 7475 raft_consensus.cc:374] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:27.823726 7475 raft_consensus.cc:729] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e005ca758f774bf2add1f97406a15853, State: Initialized, Role: FOLLOWER
I20260925 14:57:27.823880 7475 consensus_queue.cc:260] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005ca758f774bf2add1f97406a15853" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33171 } }
I20260925 14:57:27.823971 7475 raft_consensus.cc:388] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:27.824008 7475 raft_consensus.cc:482] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:27.824060 7475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:27.824132 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7408
I20260925 14:57:27.824251 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:27.824471 21276 external_mini_cluster.cc:1428] Setting key bb46f4bb0fb3cad6b75e4090d0e58cb7
I20260925 14:57:27.824715 7475 raft_consensus.cc:504] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005ca758f774bf2add1f97406a15853" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33171 } }
I20260925 14:57:27.824836 7475 leader_election.cc:302] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e005ca758f774bf2add1f97406a15853; no voters:
I20260925 14:57:27.825004 7475 leader_election.cc:288] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:27.825054 7478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:27.825206 7478 raft_consensus.cc:686] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [term 1 LEADER]: Becoming Leader. State: Replica: e005ca758f774bf2add1f97406a15853, State: Running, Role: LEADER
I20260925 14:57:27.825317 7475 sys_catalog.cc:564] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:27.825316 7478 consensus_queue.cc:237] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005ca758f774bf2add1f97406a15853" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33171 } }
I20260925 14:57:27.825796 7480 sys_catalog.cc:455] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e005ca758f774bf2add1f97406a15853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005ca758f774bf2add1f97406a15853" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33171 } } }
I20260925 14:57:27.825894 7480 sys_catalog.cc:458] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:27.825891 7482 sys_catalog.cc:455] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e005ca758f774bf2add1f97406a15853. Latest consensus state: current_term: 1 leader_uuid: "e005ca758f774bf2add1f97406a15853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e005ca758f774bf2add1f97406a15853" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 33171 } } }
I20260925 14:57:27.826037 7482 sys_catalog.cc:458] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:27.827431 7496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:27.827494 7496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:27.828321 7490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:27.828534 7490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:27.830210 7490 catalog_manager.cc:1347] Generated new cluster ID: 4fcd76f508b2486894c8bdbe83af9490
I20260925 14:57:27.830269 7490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:27.831637 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.825074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.20.199.62:33171)
0925 14:57:27.825280 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.825289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.825683 (+ 394us) server_negotiation.cc:407] Connection header received
0925 14:57:27.825830 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.825840 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.826110 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.826260 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.826496 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.827868 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.828985 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.829760 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.829889 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.830795 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.830799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.830799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:27.831005 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.831046 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:27.831061 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.831121 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.831330 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.831410 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260925 14:57:27.835098 7490 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:27.835661 7490 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:27.845430 7490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: Generated new TSK 0
I20260925 14:57:27.845638 7490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:27.850905 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:33171
--builtin_ntp_servers=127.20.199.20:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:27.933207 7499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:27.933537 7499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:27.933578 7499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:27.935760 7499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:27.935824 7499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:27.935880 7499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:27.937947 7499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:33171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:27.938211 7499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:27.938577 7499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:27.944059 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.944067 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:27.944159 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:27.944456 7499 server_base.cc:1034] running on GCE node
I20260925 14:57:27.944675 7499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:27.944917 7499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:27.946139 7499 hybrid_clock.cc:648] HybridClock initialized: now 1790348247946123 us; error 30 us; skew 500 ppm
I20260925 14:57:27.947696 7499 webserver.cc:460] Webserver started at http://127.20.199.1:39567/ using document root <none> and password file <none>
I20260925 14:57:27.947978 7499 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:27.948057 7499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:27.948217 7499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:27.949352 7499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "3ac0b6ea59d14806b5696daa08b15e94"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "5493fecdf12054c72478726ed6068b39"
server_key_iv: "f497291f2e56e06ae2fb476024e11907"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.949839 7499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "3ac0b6ea59d14806b5696daa08b15e94"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "5493fecdf12054c72478726ed6068b39"
server_key_iv: "f497291f2e56e06ae2fb476024e11907"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.951320 7499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:27.952343 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.952571 7499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:27.952695 7499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "3ac0b6ea59d14806b5696daa08b15e94"
format_stamp: "Formatted at 2026-09-25 14:57:27 on dist-test-slave-8qt6"
server_key: "5493fecdf12054c72478726ed6068b39"
server_key_iv: "f497291f2e56e06ae2fb476024e11907"
server_key_version: "encryptionkey@0"
I20260925 14:57:27.952785 7499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:27.975716 7499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:27.976085 7499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:27.976253 7499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:27.976509 7499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:27.976866 7499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:27.976920 7499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.976961 7499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:27.977011 7499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:27.982928 7499 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:46743
I20260925 14:57:27.983014 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:46743 every 8 connection(s)
I20260925 14:57:27.983349 7499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:27.987680 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7499
I20260925 14:57:27.987815 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:27.988091 21276 external_mini_cluster.cc:1428] Setting key 7eb9d4e7db0a7eed0e525844fc2ca113
I20260925 14:57:27.989147 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:33171
--builtin_ntp_servers=127.20.199.20:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:27.989425 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.984193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:50163 (local address 127.20.199.62:33171)
0925 14:57:27.984260 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:27.984262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:27.984522 (+ 260us) server_negotiation.cc:407] Connection header received
0925 14:57:27.985067 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:27.985068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:27.985083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:27.985140 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:27.985847 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.986353 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.987633 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.987942 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.988014 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.989059 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:27.989060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:27.989060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:27.989064 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:27.989078 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:27.989081 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:27.989135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:27.989269 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:57:27.989319 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:57:27.989663 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:27.984175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:33171 (local address 127.20.199.1:50163)
0925 14:57:27.984387 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:27.984421 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:27.984950 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:27.985214 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:27.985217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:27.985226 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:27.985747 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.985752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.986453 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.986454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.987528 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:27.987532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:27.988045 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:27.988046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:27.988211 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:27.988845 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:27.988846 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:27.988885 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:27.988913 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:27.989158 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:27.989164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:27.989167 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:27.989248 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:57:27.989360 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260925 14:57:27.990109 7627 heartbeater.cc:347] Connected to a master server at 127.20.199.62:33171
I20260925 14:57:27.990206 7627 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:27.990440 7627 heartbeater.cc:511] Master 127.20.199.62:33171 requested a full tablet report, sending...
I20260925 14:57:27.990989 7439 ts_manager.cc:194] Registered new tserver with Master: 3ac0b6ea59d14806b5696daa08b15e94 (127.20.199.1:46743)
I20260925 14:57:27.991614 7439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.1:50163
W20260925 14:57:28.072321 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:28.072513 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:28.072535 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:28.073757 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:28.073798 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:28.073824 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:28.075240 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:33171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:28.075481 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:28.075779 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:28.080950 7630 server_base.cc:1034] running on GCE node
W20260925 14:57:28.081079 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:28.081125 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:28.081269 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:28.081521 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:28.081807 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:28.082980 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790348248082958 us; error 39 us; skew 500 ppm
I20260925 14:57:28.084206 7630 webserver.cc:460] Webserver started at http://127.20.199.2:36163/ using document root <none> and password file <none>
I20260925 14:57:28.084471 7630 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:28.084542 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:28.084695 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:28.085862 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "7baecc3ef0b545adbd384486c271ec32"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-8qt6"
server_key: "686cfdf004e228657c9d4f85d043cdd5"
server_key_iv: "faf59c05660259340fe7028d48368060"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.086223 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "7baecc3ef0b545adbd384486c271ec32"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-8qt6"
server_key: "686cfdf004e228657c9d4f85d043cdd5"
server_key_iv: "faf59c05660259340fe7028d48368060"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.087577 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:28.088543 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.088750 7630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:28.088866 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "7baecc3ef0b545adbd384486c271ec32"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-8qt6"
server_key: "686cfdf004e228657c9d4f85d043cdd5"
server_key_iv: "faf59c05660259340fe7028d48368060"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.088969 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:28.101501 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:28.101846 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:28.102013 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:28.102293 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:28.102655 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:28.102716 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.102763 7630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:28.102818 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.109299 7630 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:42157
I20260925 14:57:28.109340 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:42157 every 8 connection(s)
I20260925 14:57:28.109768 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:28.114637 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7630
I20260925 14:57:28.114724 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:28.115021 21276 external_mini_cluster.cc:1428] Setting key 4246d7da2ec8024f56b765affa69e7ff
I20260925 14:57:28.115123 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.110607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:45523 (local address 127.20.199.62:33171)
0925 14:57:28.110702 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.110705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.110900 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:57:28.111401 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.111402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.111414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.111477 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:28.112147 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.112660 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.113634 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.113825 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.113892 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.114791 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:28.114793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:28.114793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:28.114800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:28.114817 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:28.114823 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:28.114889 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.115009 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.115040 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:57:28.115327 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.110567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:33171 (local address 127.20.199.2:45523)
0925 14:57:28.110779 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.110813 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.111306 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.111532 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.111535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.111543 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:28.112059 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.112063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.112750 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.112751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.113556 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.113558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.113911 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.113912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.114019 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.114610 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:28.114611 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:28.114640 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:28.114664 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:28.114909 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:28.114915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:28.114918 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.114977 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.115080 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260925 14:57:28.115931 7758 heartbeater.cc:347] Connected to a master server at 127.20.199.62:33171
I20260925 14:57:28.116050 7758 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:28.116025 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:33171
--builtin_ntp_servers=127.20.199.20:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:28.116278 7758 heartbeater.cc:511] Master 127.20.199.62:33171 requested a full tablet report, sending...
I20260925 14:57:28.116685 7439 ts_manager.cc:194] Registered new tserver with Master: 7baecc3ef0b545adbd384486c271ec32 (127.20.199.2:42157)
I20260925 14:57:28.117096 7439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.2:45523
W20260925 14:57:28.198129 7761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:28.198331 7761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:28.198354 7761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:28.199509 7761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:28.199553 7761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:28.199577 7761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:28.200917 7761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:33171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:28.201167 7761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:28.201541 7761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:28.206678 7761 server_base.cc:1034] running on GCE node
W20260925 14:57:28.206583 7766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:28.206589 7767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:28.206635 7769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:28.207170 7761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:28.207438 7761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:28.208608 7761 hybrid_clock.cc:648] HybridClock initialized: now 1790348248208595 us; error 41 us; skew 500 ppm
I20260925 14:57:28.209981 7761 webserver.cc:460] Webserver started at http://127.20.199.3:44747/ using document root <none> and password file <none>
I20260925 14:57:28.210247 7761 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:28.210320 7761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:28.210484 7761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:28.211566 7761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "5d9c188d8d4446498d364e468e35d5bd"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-8qt6"
server_key: "f320fa5ca9092f387e78bc0c40845725"
server_key_iv: "a1b06be20b8f5df0f7758a521670315d"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.211937 7761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "5d9c188d8d4446498d364e468e35d5bd"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-8qt6"
server_key: "f320fa5ca9092f387e78bc0c40845725"
server_key_iv: "a1b06be20b8f5df0f7758a521670315d"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.213263 7761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:28.214148 7775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.214330 7761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:28.214429 7761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "5d9c188d8d4446498d364e468e35d5bd"
format_stamp: "Formatted at 2026-09-25 14:57:28 on dist-test-slave-8qt6"
server_key: "f320fa5ca9092f387e78bc0c40845725"
server_key_iv: "a1b06be20b8f5df0f7758a521670315d"
server_key_version: "encryptionkey@0"
I20260925 14:57:28.214524 7761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:28.241856 7761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:28.242223 7761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:28.242393 7761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:28.242653 7761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:28.243013 7761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:28.243068 7761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.243105 7761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:28.243132 7761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:28.248880 7761 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:33629
I20260925 14:57:28.248927 7888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:33629 every 8 connection(s)
I20260925 14:57:28.249274 7761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:28.251200 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7761
I20260925 14:57:28.251322 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:28.251554 21276 external_mini_cluster.cc:1428] Setting key d90ad07683230512545296266aae7d0f
I20260925 14:57:28.254788 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.250038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:35585 (local address 127.20.199.62:33171)
0925 14:57:28.250103 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.250106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.250342 (+ 236us) server_negotiation.cc:407] Connection header received
0925 14:57:28.250819 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.250820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.250830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.250888 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:28.251833 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.252529 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.253555 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.253736 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.253789 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.254570 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:28.254571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:28.254571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:28.254576 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:28.254589 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:28.254593 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:28.254636 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.254708 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.254731 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:57:28.255066 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.250005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:33171 (local address 127.20.199.3:35585)
0925 14:57:28.250210 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.250251 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.250735 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.250953 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.250956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.250964 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:28.251726 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.251733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.252631 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.252632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.253479 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.253482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.253807 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.253808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.253946 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.254440 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:28.254441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:28.254469 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:28.254492 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:28.254647 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:28.254650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:28.254652 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.254711 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.254789 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260925 14:57:28.255690 7889 heartbeater.cc:347] Connected to a master server at 127.20.199.62:33171
I20260925 14:57:28.255800 7889 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:28.256026 7889 heartbeater.cc:511] Master 127.20.199.62:33171 requested a full tablet report, sending...
I20260925 14:57:28.256381 7439 ts_manager.cc:194] Registered new tserver with Master: 5d9c188d8d4446498d364e468e35d5bd (127.20.199.3:33629)
I20260925 14:57:28.256735 7439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.3:35585
I20260925 14:57:28.263422 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:28.267729 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.264738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.20.199.62:33171)
0925 14:57:28.264818 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.264820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.264916 (+ 96us) server_negotiation.cc:407] Connection header received
0925 14:57:28.264956 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.264956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.264968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.265015 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:28.265154 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.265705 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.266567 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.266763 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.266869 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.267482 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:28.267483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:28.267483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:28.267487 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:28.267502 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:28.267506 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:28.267593 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.267652 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.267662 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:57:28.269547 7439 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:28.271057 7439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:28.277778 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.273195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:42157 (local address 127.0.0.1:44330)
0925 14:57:28.273358 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.273396 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.273482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.273974 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.273975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.273977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.274028 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.274030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.275360 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.275360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.276517 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.276519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.277433 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.277434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.277516 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.277533 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.277615 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.277672 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":103,"spinlock_wait_cycles":896,"thread_start_us":107,"threads_started":1}
I20260925 14:57:28.277921 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.273604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.20.199.1:46743)
0925 14:57:28.273775 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.273778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.273787 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:28.273817 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.273819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.273830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.273921 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.274207 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.275265 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.276817 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.277542 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.277624 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.277706 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.277777 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.277822 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260925 14:57:28.277935 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.273196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:46743 (local address 127.0.0.1:58662)
0925 14:57:28.273358 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.273396 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.273482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.273923 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.273925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.273927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.273997 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.274000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.275377 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.275378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.276698 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.276701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.277634 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.277635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.277695 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.277699 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.277745 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.277789 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":63,"thread_start_us":58,"threads_started":1}
I20260925 14:57:28.278362 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.273392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.20.199.2:42157)
0925 14:57:28.273808 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.273811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.273820 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:28.273854 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.273856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.273869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.273975 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.274107 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.275260 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.276603 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.277313 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.277390 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.277521 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.278103 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.278154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":93,"threads_started":1}
I20260925 14:57:28.278683 7561 tablet_service.cc:1467] Processing CreateTablet for tablet d57a9ea0d04f486488de7dd4387f5f31 (DEFAULT_TABLE table=test-table [id=bc911e0ef0dc487bb2e338f0ad0fbc2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:28.278935 7692 tablet_service.cc:1467] Processing CreateTablet for tablet d57a9ea0d04f486488de7dd4387f5f31 (DEFAULT_TABLE table=test-table [id=bc911e0ef0dc487bb2e338f0ad0fbc2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:28.279088 7561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57a9ea0d04f486488de7dd4387f5f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:28.279265 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57a9ea0d04f486488de7dd4387f5f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:28.281607 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.273469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.20.199.3:33629)
0925 14:57:28.273637 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.273643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.274196 (+ 553us) server_negotiation.cc:407] Connection header received
0925 14:57:28.274244 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.274246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.274263 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.274340 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.274531 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.275223 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.276795 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.277565 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.277659 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.277749 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.278581 (+ 832us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.281510 (+ 2929us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260925 14:57:28.278388 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.273208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33629 (local address 127.0.0.1:44746)
0925 14:57:28.274068 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.274079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.274128 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.274413 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.274414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.274415 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.274445 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.274447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.275340 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.275341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.276698 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.276701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.277780 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.277780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.277833 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.277836 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.278261 (+ 425us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.278310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":144,"thread_start_us":112,"threads_started":1}
I20260925 14:57:28.282215 7904 tablet_bootstrap.cc:492] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: Bootstrap starting.
I20260925 14:57:28.282397 7823 tablet_service.cc:1467] Processing CreateTablet for tablet d57a9ea0d04f486488de7dd4387f5f31 (DEFAULT_TABLE table=test-table [id=bc911e0ef0dc487bb2e338f0ad0fbc2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:28.282717 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57a9ea0d04f486488de7dd4387f5f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:28.282920 7904 tablet_bootstrap.cc:654] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:28.283221 7904 log.cc:824] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:28.283601 7903 tablet_bootstrap.cc:492] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Bootstrap starting.
I20260925 14:57:28.284152 7903 tablet_bootstrap.cc:654] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:28.284399 7903 log.cc:824] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:28.284976 7904 tablet_bootstrap.cc:492] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: No bootstrap required, opened a new log
I20260925 14:57:28.285048 7904 ts_tablet_manager.cc:1397] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:57:28.285127 7903 tablet_bootstrap.cc:492] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: No bootstrap required, opened a new log
I20260925 14:57:28.285202 7903 ts_tablet_manager.cc:1397] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:28.285753 7907 tablet_bootstrap.cc:492] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Bootstrap starting.
I20260925 14:57:28.286296 7907 tablet_bootstrap.cc:654] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:28.286586 7907 log.cc:824] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:28.287221 7907 tablet_bootstrap.cc:492] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: No bootstrap required, opened a new log
I20260925 14:57:28.287299 7907 ts_tablet_manager.cc:1397] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:28.287891 7903 raft_consensus.cc:348] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.287891 7904 raft_consensus.cc:348] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.288072 7903 raft_consensus.cc:374] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:28.288071 7904 raft_consensus.cc:374] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:28.288111 7904 raft_consensus.cc:729] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac0b6ea59d14806b5696daa08b15e94, State: Initialized, Role: FOLLOWER
I20260925 14:57:28.288111 7903 raft_consensus.cc:729] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7baecc3ef0b545adbd384486c271ec32, State: Initialized, Role: FOLLOWER
I20260925 14:57:28.288257 7903 consensus_queue.cc:260] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.288257 7904 consensus_queue.cc:260] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.288539 7904 ts_tablet_manager.cc:1428] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 14:57:28.288539 7903 ts_tablet_manager.cc:1428] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:57:28.288671 7758 heartbeater.cc:503] Master 127.20.199.62:33171 was elected leader, sending a full tablet report...
I20260925 14:57:28.288987 7627 heartbeater.cc:503] Master 127.20.199.62:33171 was elected leader, sending a full tablet report...
I20260925 14:57:28.289678 7907 raft_consensus.cc:348] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.289853 7907 raft_consensus.cc:374] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:28.289901 7907 raft_consensus.cc:729] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9c188d8d4446498d364e468e35d5bd, State: Initialized, Role: FOLLOWER
I20260925 14:57:28.290014 7907 consensus_queue.cc:260] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.290258 7907 ts_tablet_manager.cc:1428] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:28.290352 7889 heartbeater.cc:503] Master 127.20.199.62:33171 was elected leader, sending a full tablet report...
W20260925 14:57:28.360384 7759 tablet.cc:2369] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:28.484134 7628 tablet.cc:2369] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:28.500036 7890 tablet.cc:2369] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:28.701293 7909 raft_consensus.cc:482] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:28.701458 7909 raft_consensus.cc:504] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.701910 7909 leader_election.cc:288] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac0b6ea59d14806b5696daa08b15e94 (127.20.199.1:46743), 5d9c188d8d4446498d364e468e35d5bd (127.20.199.3:33629)
I20260925 14:57:28.704351 7911 raft_consensus.cc:482] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:28.704460 7911 raft_consensus.cc:504] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.705132 7911 leader_election.cc:288] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7baecc3ef0b545adbd384486c271ec32 (127.20.199.2:42157), 3ac0b6ea59d14806b5696daa08b15e94 (127.20.199.1:46743)
I20260925 14:57:28.705663 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.702244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:39671 (local address 127.20.199.1:46743)
0925 14:57:28.702287 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.702289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.702321 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:57:28.702425 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.702426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.702436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.702533 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.702688 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.703296 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.704704 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.705262 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.705324 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.705334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.705547 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.705595 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:28.705734 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.702213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:46299 (local address 127.20.199.3:33629)
0925 14:57:28.702264 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.702267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.702362 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:57:28.702424 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.702426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.702437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.702520 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.702690 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.703239 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.704693 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.705263 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.705325 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.705334 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.705635 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.705674 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:57:28.705951 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.702127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:46743 (local address 127.20.199.2:39671)
0925 14:57:28.702259 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.702271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.702339 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.702542 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.702543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.702545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.702617 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.702619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.703373 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.703373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.704556 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.704558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.705348 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.705349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.705444 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.705448 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.705777 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.705866 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":31,"thread_start_us":84,"threads_started":1}
I20260925 14:57:28.706283 7581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a9ea0d04f486488de7dd4387f5f31" candidate_uuid: "7baecc3ef0b545adbd384486c271ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac0b6ea59d14806b5696daa08b15e94" is_pre_election: true
I20260925 14:57:28.706435 7581 raft_consensus.cc:2454] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7baecc3ef0b545adbd384486c271ec32 in term 0.
I20260925 14:57:28.706727 7648 leader_election.cc:302] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac0b6ea59d14806b5696daa08b15e94, 7baecc3ef0b545adbd384486c271ec32; no voters:
I20260925 14:57:28.706835 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.702132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:33629 (local address 127.20.199.2:46299)
0925 14:57:28.702290 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.702299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.702364 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.702525 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.702526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.702528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.702606 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.702609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.703331 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.703332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.704556 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.704558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.705435 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.705436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.705536 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.705542 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.706652 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.706698 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260925 14:57:28.706871 7909 raft_consensus.cc:2781] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:28.706913 7909 raft_consensus.cc:482] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:28.706935 7909 raft_consensus.cc:3037] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:28.707082 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a9ea0d04f486488de7dd4387f5f31" candidate_uuid: "7baecc3ef0b545adbd384486c271ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9c188d8d4446498d364e468e35d5bd" is_pre_election: true
I20260925 14:57:28.707159 7843 raft_consensus.cc:2454] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7baecc3ef0b545adbd384486c271ec32 in term 0.
I20260925 14:57:28.707839 7909 raft_consensus.cc:504] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.708102 7581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a9ea0d04f486488de7dd4387f5f31" candidate_uuid: "7baecc3ef0b545adbd384486c271ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac0b6ea59d14806b5696daa08b15e94"
I20260925 14:57:28.708187 7581 raft_consensus.cc:3037] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:28.708232 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a9ea0d04f486488de7dd4387f5f31" candidate_uuid: "7baecc3ef0b545adbd384486c271ec32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9c188d8d4446498d364e468e35d5bd"
I20260925 14:57:28.708304 7843 raft_consensus.cc:3037] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:28.708717 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.704946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:46743 (local address 127.20.199.3:57801)
0925 14:57:28.704987 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.705007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.705039 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.705639 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.705640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.705642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.705696 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.705698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.706572 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.706573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.707690 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.707694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.708491 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.708492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.708577 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.708579 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.708609 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.708655 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260925 14:57:28.708746 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.705170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:56967 (local address 127.20.199.2:42157)
0925 14:57:28.705208 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.705210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.705299 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:57:28.705387 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.705388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.705420 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.705450 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.705653 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.706358 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.707864 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.708372 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.708419 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.708426 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.708658 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.708690 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:28.708969 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.705372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:57801 (local address 127.20.199.1:46743)
0925 14:57:28.705502 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.705504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.705511 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:28.705531 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.705532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.705541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.705763 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:28.705793 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.706503 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.707792 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.708386 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.708816 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.708827 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.708846 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.708897 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260925 14:57:28.709040 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.704982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:42157 (local address 127.20.199.3:56967)
0925 14:57:28.705241 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.705249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:28.705305 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:28.705505 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:28.705506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:28.705508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:28.705574 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.705575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.706454 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.706455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.707764 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:28.707767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.708493 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.708493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:28.708577 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.708580 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:28.708901 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 14:57:28.708940 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":14,"thread_start_us":84,"threads_started":1}
I20260925 14:57:28.709101 7843 raft_consensus.cc:2454] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7baecc3ef0b545adbd384486c271ec32 in term 1.
I20260925 14:57:28.709101 7581 raft_consensus.cc:2454] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7baecc3ef0b545adbd384486c271ec32 in term 1.
I20260925 14:57:28.709316 7648 leader_election.cc:302] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac0b6ea59d14806b5696daa08b15e94, 7baecc3ef0b545adbd384486c271ec32; no voters:
I20260925 14:57:28.709326 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a9ea0d04f486488de7dd4387f5f31" candidate_uuid: "5d9c188d8d4446498d364e468e35d5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7baecc3ef0b545adbd384486c271ec32" is_pre_election: true
I20260925 14:57:28.709450 7581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a9ea0d04f486488de7dd4387f5f31" candidate_uuid: "5d9c188d8d4446498d364e468e35d5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac0b6ea59d14806b5696daa08b15e94" is_pre_election: true
I20260925 14:57:28.709502 7712 raft_consensus.cc:2379] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d9c188d8d4446498d364e468e35d5bd in current term 1: Already voted for candidate 7baecc3ef0b545adbd384486c271ec32 in this term.
I20260925 14:57:28.709519 7581 raft_consensus.cc:2379] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d9c188d8d4446498d364e468e35d5bd in current term 1: Already voted for candidate 7baecc3ef0b545adbd384486c271ec32 in this term.
I20260925 14:57:28.709651 7909 leader_election.cc:288] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [CANDIDATE]: Term 1 election: Requested vote from peers 3ac0b6ea59d14806b5696daa08b15e94 (127.20.199.1:46743), 5d9c188d8d4446498d364e468e35d5bd (127.20.199.3:33629)
I20260925 14:57:28.709744 7909 raft_consensus.cc:2781] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:28.709789 7909 raft_consensus.cc:686] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 1 LEADER]: Becoming Leader. State: Replica: 7baecc3ef0b545adbd384486c271ec32, State: Running, Role: LEADER
I20260925 14:57:28.709792 7778 leader_election.cc:302] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d9c188d8d4446498d364e468e35d5bd; no voters: 3ac0b6ea59d14806b5696daa08b15e94, 7baecc3ef0b545adbd384486c271ec32
I20260925 14:57:28.709872 7909 consensus_queue.cc:237] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } }
I20260925 14:57:28.709918 7911 raft_consensus.cc:2726] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 14:57:28.710641 7437 catalog_manager.cc:5495] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7baecc3ef0b545adbd384486c271ec32 (127.20.199.2). New cstate: current_term: 1 leader_uuid: "7baecc3ef0b545adbd384486c271ec32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baecc3ef0b545adbd384486c271ec32" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 42157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 } health_report { overall_health: UNKNOWN } } }
I20260925 14:57:28.718314 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.715766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.20.199.2:42157)
0925 14:57:28.715819 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:28.715821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:28.715849 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:57:28.715875 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:28.715876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:28.715890 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:28.715940 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:28.716059 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.716494 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.717290 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:28.717555 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:28.717647 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:28.717891 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:28.718027 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:28.718160 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:28.718222 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:57:28.718235 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:57:28.720017 7581 raft_consensus.cc:1261] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94 [term 1 FOLLOWER]: Refusing update from remote peer 7baecc3ef0b545adbd384486c271ec32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:28.720031 7843 raft_consensus.cc:1261] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 1 FOLLOWER]: Refusing update from remote peer 7baecc3ef0b545adbd384486c271ec32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:28.720310 7909 consensus_queue.cc:1035] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac0b6ea59d14806b5696daa08b15e94" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:28.720480 7916 consensus_queue.cc:1035] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9c188d8d4446498d364e468e35d5bd" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:28.723096 7919 mvcc.cc:205] Tried to move back new op lower bound from 7333266426754777088 to 7333266426716229632. Current Snapshot: MvccSnapshot[applied={T|T < 7333266426754777088}]
I20260925 14:57:28.723899 7920 mvcc.cc:205] Tried to move back new op lower bound from 7333266426754777088 to 7333266426716229632. Current Snapshot: MvccSnapshot[applied={T|T < 7333266426754777088 or (T in {7333266426754777088})}]
I20260925 14:57:28.725255 7438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:28.725425 7438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:28.726226 7438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e005ca758f774bf2add1f97406a15853: Sending DeleteTablet for 3 replicas of tablet d57a9ea0d04f486488de7dd4387f5f31
I20260925 14:57:28.726575 7692 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a9ea0d04f486488de7dd4387f5f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:28 UTC) from {username='slave'} at 127.0.0.1:44330
I20260925 14:57:28.726632 7561 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a9ea0d04f486488de7dd4387f5f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:28 UTC) from {username='slave'} at 127.0.0.1:58662
I20260925 14:57:28.726727 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a9ea0d04f486488de7dd4387f5f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:28 UTC) from {username='slave'} at 127.0.0.1:44746
I20260925 14:57:28.726802 7929 tablet_replica.cc:330] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: stopping tablet replica
I20260925 14:57:28.726902 7929 raft_consensus.cc:2229] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:28.727005 7929 raft_consensus.cc:2258] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:28.727020 7930 tablet_replica.cc:330] T d57a9ea0d04f486488de7dd4387f5f31 P 3ac0b6ea59d14806b5696daa08b15e94: stopping tablet replica
I20260925 14:57:28.727058 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7499
I20260925 14:57:28.728211 7929 ts_tablet_manager.cc:1905] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:28.729084 7931 tablet_replica.cc:330] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: stopping tablet replica
I20260925 14:57:28.729215 7931 raft_consensus.cc:2229] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:28.729328 7931 raft_consensus.cc:2258] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:28.730059 7929 ts_tablet_manager.cc:1918] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:28.730136 7929 log.cc:1196] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/d57a9ea0d04f486488de7dd4387f5f31
I20260925 14:57:28.730381 7929 ts_tablet_manager.cc:1939] T d57a9ea0d04f486488de7dd4387f5f31 P 7baecc3ef0b545adbd384486c271ec32: Deleting consensus metadata
I20260925 14:57:28.730808 7931 ts_tablet_manager.cc:1905] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:28.731056 7425 catalog_manager.cc:4841] TS 7baecc3ef0b545adbd384486c271ec32 (127.20.199.2:42157): tablet d57a9ea0d04f486488de7dd4387f5f31 (table test-table [id=bc911e0ef0dc487bb2e338f0ad0fbc2c]) successfully deleted
I20260925 14:57:28.732391 7931 ts_tablet_manager.cc:1918] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:28.732467 7931 log.cc:1196] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/d57a9ea0d04f486488de7dd4387f5f31
I20260925 14:57:28.732714 7931 ts_tablet_manager.cc:1939] T d57a9ea0d04f486488de7dd4387f5f31 P 5d9c188d8d4446498d364e468e35d5bd: Deleting consensus metadata
W20260925 14:57:28.733029 7426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:46743 (error 108)
I20260925 14:57:28.733094 7424 catalog_manager.cc:4841] TS 5d9c188d8d4446498d364e468e35d5bd (127.20.199.3:33629): tablet d57a9ea0d04f486488de7dd4387f5f31 (table test-table [id=bc911e0ef0dc487bb2e338f0ad0fbc2c]) successfully deleted
I20260925 14:57:28.733366 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7630
I20260925 14:57:28.734069 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:28.733452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:46743 (local address 127.0.0.1:58664)
0925 14:57:28.733984 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:28.734028 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509}
W20260925 14:57:28.734179 7426 catalog_manager.cc:4584] TS 3ac0b6ea59d14806b5696daa08b15e94 (127.20.199.1:46743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d57a9ea0d04f486488de7dd4387f5f31: Network error: Client connection negotiation failed: client connection to 127.20.199.1:46743: connect: Connection refused (error 111)
I20260925 14:57:28.739333 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7761
I20260925 14:57:28.744735 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7408
2026-09-25T14:57:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:28 dist-test-slave-8qt6 krb5kdc[7935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:28 dist-test-slave-8qt6 krb5kdc[7935](info): set up 2 sockets
Sep 25 14:57:28 dist-test-slave-8qt6 krb5kdc[7935](info): commencing operation
krb5kdc: starting...
W20260925 14:57:29.793182 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:29 dist-test-slave-8qt6 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:29.912207 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37903
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:37903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:29.993635 7948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:29.993878 7948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:29.993918 7948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:29.995085 7948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:29.995154 7948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:29.995182 7948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:29.995214 7948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:29.995239 7948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:29.996819 7948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:37903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:37903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:29.997090 7948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:29.997313 7948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:30.002020 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.002067 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.002035 7956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.002461 7948 server_base.cc:1034] running on GCE node
I20260925 14:57:30.002719 7948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:30.003006 7948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:30.004153 7948 hybrid_clock.cc:648] HybridClock initialized: now 1790348250004128 us; error 38 us; skew 500 ppm
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:30.014732 7948 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:30.015239 7948 webserver.cc:460] Webserver started at http://127.20.199.62:33559/ using document root <none> and password file <none>
I20260925 14:57:30.015489 7948 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:30.015558 7948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:30.015704 7948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:30.016829 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "d5f17f74119543d4b39b7cb9e60e2df3"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "b322daa548bc22fc09ef694d0968df6f"
server_key_iv: "90b61ce6e4b40f832329dca4cc351409"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.017195 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "d5f17f74119543d4b39b7cb9e60e2df3"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "b322daa548bc22fc09ef694d0968df6f"
server_key_iv: "90b61ce6e4b40f832329dca4cc351409"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.018661 7948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:30.019479 7963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.019696 7948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:30.019804 7948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "d5f17f74119543d4b39b7cb9e60e2df3"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "b322daa548bc22fc09ef694d0968df6f"
server_key_iv: "90b61ce6e4b40f832329dca4cc351409"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.019876 7948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:30.054498 7948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:30.057030 7948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:30.057193 7948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:30.061184 7948 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:37903
I20260925 14:57:30.061224 8015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:37903 every 8 connection(s)
I20260925 14:57:30.061625 7948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:30.062484 8016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:30.064842 8016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3: Bootstrap starting.
I20260925 14:57:30.065366 8016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:30.065768 8016 log.cc:824] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:30.066447 8016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3: No bootstrap required, opened a new log
I20260925 14:57:30.067021 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 7948
I20260925 14:57:30.067149 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:30.067334 21276 external_mini_cluster.cc:1428] Setting key 9908f08f629608d623c543672342f545
I20260925 14:57:30.068544 8016 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37903 } }
I20260925 14:57:30.068717 8016 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:30.068754 8016 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f17f74119543d4b39b7cb9e60e2df3, State: Initialized, Role: FOLLOWER
I20260925 14:57:30.068874 8016 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37903 } }
I20260925 14:57:30.068955 8016 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:30.068979 8016 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:30.069044 8016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:30.070132 8016 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37903 } }
I20260925 14:57:30.070238 8016 leader_election.cc:302] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f17f74119543d4b39b7cb9e60e2df3; no voters:
I20260925 14:57:30.070380 8016 leader_election.cc:288] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:30.070421 8021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:30.070590 8021 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [term 1 LEADER]: Becoming Leader. State: Replica: d5f17f74119543d4b39b7cb9e60e2df3, State: Running, Role: LEADER
I20260925 14:57:30.070621 8016 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:30.070684 8021 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37903 } }
I20260925 14:57:30.070981 8021 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f17f74119543d4b39b7cb9e60e2df3. Latest consensus state: current_term: 1 leader_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37903 } } }
I20260925 14:57:30.071074 8021 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:30.070989 8023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f17f74119543d4b39b7cb9e60e2df3" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 37903 } } }
I20260925 14:57:30.071206 8023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:30.071372 8028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:30.071899 8028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:30.073613 8028 catalog_manager.cc:1347] Generated new cluster ID: 08598c1d05bc404b8f78d211a59387cb
I20260925 14:57:30.073669 8028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:30.078496 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.067878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.20.199.62:37903)
0925 14:57:30.068127 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.068136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.068172 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:57:30.068510 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.068518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.068580 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.068730 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.069463 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.070874 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.072080 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.072683 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.072790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.075407 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.075411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.075412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:30.075423 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.076959 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.077311 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.077315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.077316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.077463 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.077661 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.077662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.077663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.077923 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.078091 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.078189 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.078337 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260925 14:57:30.081799 8028 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:30.082518 8028 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:30.089627 8028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3: Generated new TSK 0
I20260925 14:57:30.089802 8028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.199.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
Entry for principal kudu/127.20.199.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab.
I20260925 14:57:30.115973 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37903
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:30.197144 8042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:30.197371 8042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:30.197412 8042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:30.198732 8042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:30.198794 8042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:30.198837 8042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:30.200294 8042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.1.keytab
--principal=kudu/127.20.199.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:30.200555 8042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:30.200779 8042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:30.206527 8050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.206530 8047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.206733 8042 server_base.cc:1034] running on GCE node
W20260925 14:57:30.206543 8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.207031 8042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:30.207266 8042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:30.208403 8042 hybrid_clock.cc:648] HybridClock initialized: now 1790348250208388 us; error 43 us; skew 500 ppm
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:30.210939 8042 init.cc:377] Logged in from keytab as kudu/127.20.199.1@KRBTEST.COM (short username kudu)
I20260925 14:57:30.211360 8042 webserver.cc:460] Webserver started at http://127.20.199.1:39073/ using document root <none> and password file <none>
I20260925 14:57:30.211596 8042 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:30.211642 8042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:30.211781 8042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:30.212796 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "aea9b36558a7477daddb8b55a7a6d163"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "f52cd9e662341935a0b9b070955f955f"
server_key_iv: "c7c841bfe005c30269d288df1dcc5dc5"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.213169 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "aea9b36558a7477daddb8b55a7a6d163"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "f52cd9e662341935a0b9b070955f955f"
server_key_iv: "c7c841bfe005c30269d288df1dcc5dc5"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.214587 8042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:30.215387 8057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.215713 8042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:30.215811 8042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "aea9b36558a7477daddb8b55a7a6d163"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "f52cd9e662341935a0b9b070955f955f"
server_key_iv: "c7c841bfe005c30269d288df1dcc5dc5"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.215880 8042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:30.245322 8042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:30.247287 8042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:30.247437 8042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:30.247689 8042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:30.248006 8042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:30.248042 8042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.248063 8042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:30.248108 8042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.253867 8042 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:36925
I20260925 14:57:30.253918 8170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:36925 every 8 connection(s)
I20260925 14:57:30.254227 8042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.1@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:30.261150 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8042
I20260925 14:57:30.261276 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:30.261572 21276 external_mini_cluster.cc:1428] Setting key df06f3cc481e331f8a939a5abf75bf75
I20260925 14:57:30.263932 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.255073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:38297 (local address 127.20.199.62:37903)
0925 14:57:30.255124 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.255126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.255317 (+ 191us) server_negotiation.cc:407] Connection header received
0925 14:57:30.255848 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.255849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.255860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.255924 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.256863 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.257371 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.258401 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.258626 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.258698 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.260703 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.260704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.260704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:30.260714 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.262411 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.262824 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.262825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.262826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.262881 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.263147 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.263147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.263148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.263260 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.263345 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.263718 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.263813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:30.263952 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.255040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37903 (local address 127.20.199.1:38297)
0925 14:57:30.255181 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.255226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.255736 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.255994 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.255998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.256562 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:30.256747 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.256752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.257515 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.257517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.258300 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.258302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.258719 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.258720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.258840 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.259260 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:30.259268 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:30.260605 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:30.262506 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:30.262513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:30.262515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:30.262733 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:30.262962 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:30.262963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:30.262963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:30.263069 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:30.263377 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:30.263385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:30.263573 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.263672 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.263795 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260925 14:57:30.264693 8171 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37903
I20260925 14:57:30.264791 8171 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:30.264974 8171 heartbeater.cc:511] Master 127.20.199.62:37903 requested a full tablet report, sending...
I20260925 14:57:30.265568 7980 ts_manager.cc:194] Registered new tserver with Master: aea9b36558a7477daddb8b55a7a6d163 (127.20.199.1:36925)
I20260925 14:57:30.266184 7980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.1@KRBTEST.COM'} at 127.20.199.1:38297
WARNING: no policy specified for kudu/127.20.199.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
Entry for principal kudu/127.20.199.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab.
I20260925 14:57:30.289738 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37903
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:30.371477 8176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:30.371722 8176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:30.371749 8176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:30.372936 8176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:30.372975 8176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:30.373001 8176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:30.374466 8176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.2.keytab
--principal=kudu/127.20.199.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:30.374642 8176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:30.374811 8176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:30.379813 8181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.379864 8176 server_base.cc:1034] running on GCE node
W20260925 14:57:30.379835 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.379832 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.380151 8176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:30.380443 8176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:30.381598 8176 hybrid_clock.cc:648] HybridClock initialized: now 1790348250381582 us; error 25 us; skew 500 ppm
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:30.383988 8176 init.cc:377] Logged in from keytab as kudu/127.20.199.2@KRBTEST.COM (short username kudu)
I20260925 14:57:30.384470 8176 webserver.cc:460] Webserver started at http://127.20.199.2:39715/ using document root <none> and password file <none>
I20260925 14:57:30.384701 8176 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:30.384771 8176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:30.384909 8176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:30.386029 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "50fececfec014cd493e9e4fd8742884f"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "9e17aa2f9a97f32a6a1a72732884ebaa"
server_key_iv: "82e075abc945a71494b625812b54d216"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.386404 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "50fececfec014cd493e9e4fd8742884f"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "9e17aa2f9a97f32a6a1a72732884ebaa"
server_key_iv: "82e075abc945a71494b625812b54d216"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.387722 8176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:30.388393 8191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.388554 8176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:30.388649 8176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "50fececfec014cd493e9e4fd8742884f"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "9e17aa2f9a97f32a6a1a72732884ebaa"
server_key_iv: "82e075abc945a71494b625812b54d216"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.388731 8176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:30.417774 8176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:30.419782 8176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:30.419941 8176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:30.420197 8176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:30.420521 8176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:30.420575 8176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.420621 8176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:30.420644 8176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.426942 8176 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:39803
I20260925 14:57:30.426995 8304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:39803 every 8 connection(s)
I20260925 14:57:30.427309 8176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.2@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:30.434993 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8176
I20260925 14:57:30.435111 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:30.435344 21276 external_mini_cluster.cc:1428] Setting key b43d8005b0bdd9004030585902aec180
I20260925 14:57:30.436264 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.428008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:51841 (local address 127.20.199.62:37903)
0925 14:57:30.428055 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.428057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.428290 (+ 233us) server_negotiation.cc:407] Connection header received
0925 14:57:30.428743 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.428744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.428759 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.428812 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.429758 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.430241 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.431221 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.431402 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.431455 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.433385 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.433386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.433386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:30.433392 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.434687 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.435193 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.435194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.435194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.435258 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.435553 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.435554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.435554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.435716 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.435785 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.436100 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.436191 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:30.436542 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.428016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37903 (local address 127.20.199.2:51841)
0925 14:57:30.428162 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.428197 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.428658 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.428865 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.428868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.429452 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:30.429674 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.429679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.430318 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.430319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.431145 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.431147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.431475 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.431476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.431577 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.431999 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:30.432007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:30.433291 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:30.434788 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:30.434795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:30.434797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:30.435114 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:30.435327 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:30.435328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:30.435328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:30.435476 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:30.435793 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:30.435798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:30.435925 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.436068 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.436254 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260925 14:57:30.437074 8305 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37903
I20260925 14:57:30.437173 8305 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:30.437374 8305 heartbeater.cc:511] Master 127.20.199.62:37903 requested a full tablet report, sending...
I20260925 14:57:30.437819 7980 ts_manager.cc:194] Registered new tserver with Master: 50fececfec014cd493e9e4fd8742884f (127.20.199.2:39803)
I20260925 14:57:30.438198 7980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.2@KRBTEST.COM'} at 127.20.199.2:51841
WARNING: no policy specified for kudu/127.20.199.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
Entry for principal kudu/127.20.199.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab.
I20260925 14:57:30.460855 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37903
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:30.541762 8310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:30.542001 8310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:30.542042 8310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:30.543254 8310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:30.543327 8310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:30.543370 8310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:30.544879 8310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.3.keytab
--principal=kudu/127.20.199.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:37903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:30.545158 8310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:30.545392 8310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:30.550349 8318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.550346 8315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.550381 8316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.550702 8310 server_base.cc:1034] running on GCE node
I20260925 14:57:30.550931 8310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:30.551170 8310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:30.552307 8310 hybrid_clock.cc:648] HybridClock initialized: now 1790348250552286 us; error 36 us; skew 500 ppm
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:30.554738 8310 init.cc:377] Logged in from keytab as kudu/127.20.199.3@KRBTEST.COM (short username kudu)
I20260925 14:57:30.555231 8310 webserver.cc:460] Webserver started at http://127.20.199.3:37845/ using document root <none> and password file <none>
I20260925 14:57:30.555426 8310 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:30.555466 8310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:30.555555 8310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:30.556545 8310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "cbb1374e57854db9887e3a920cddb46d"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "057833ca82ad65d5296d31e9c96b5ebc"
server_key_iv: "f67422ce219fa44ac19362290a96c447"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.556841 8310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "cbb1374e57854db9887e3a920cddb46d"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "057833ca82ad65d5296d31e9c96b5ebc"
server_key_iv: "f67422ce219fa44ac19362290a96c447"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.558214 8310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:57:30.558897 8325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.559077 8310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:30.559146 8310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "cbb1374e57854db9887e3a920cddb46d"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "057833ca82ad65d5296d31e9c96b5ebc"
server_key_iv: "f67422ce219fa44ac19362290a96c447"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.559237 8310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:30.585623 8310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:30.587756 8310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:30.587905 8310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:30.588188 8310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:30.588503 8310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:30.588537 8310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.588562 8310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:30.588608 8310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.594887 8310 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:37157
I20260925 14:57:30.594929 8438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:37157 every 8 connection(s)
I20260925 14:57:30.595249 8310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:30.595564 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8310
I20260925 14:57:30.595669 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:30.595855 21276 external_mini_cluster.cc:1428] Setting key 2f5219e0a8874fff03471bc3e3417496
Sep 25 14:57:30 dist-test-slave-8qt6 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.3@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:30.604604 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.596029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:59677 (local address 127.20.199.62:37903)
0925 14:57:30.596091 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.596093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.596343 (+ 250us) server_negotiation.cc:407] Connection header received
0925 14:57:30.597015 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.597016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.597028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.597073 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.598077 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.598594 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.599642 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.599852 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.599938 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.601985 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.601987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.601987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:30.602003 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.603229 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.603653 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.603654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.603654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.603713 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.603952 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.603953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.603953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.604076 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.604154 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.604428 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.604509 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:57:30.604790 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.596012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:37903 (local address 127.20.199.3:59677)
0925 14:57:30.596223 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.596273 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.596916 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.597142 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.597146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.597771 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:30.597966 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.597970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.598689 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.598690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.599554 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.599556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.599947 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.599948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.600083 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.600537 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:30.600546 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:30.601893 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:30.603314 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:30.603320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:30.603323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:30.603566 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:30.603779 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:30.603780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:30.603781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:30.603883 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:30.604147 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:30.604153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:30.604292 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.604393 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.604558 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260925 14:57:30.605322 8439 heartbeater.cc:347] Connected to a master server at 127.20.199.62:37903
I20260925 14:57:30.605439 8439 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:30.605623 8439 heartbeater.cc:511] Master 127.20.199.62:37903 requested a full tablet report, sending...
I20260925 14:57:30.606003 7980 ts_manager.cc:194] Registered new tserver with Master: cbb1374e57854db9887e3a920cddb46d (127.20.199.3:37157)
I20260925 14:57:30.606389 7980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.199.3@KRBTEST.COM'} at 127.20.199.3:59677
I20260925 14:57:30.607518 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:30.614149 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.608779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.20.199.62:37903)
0925 14:57:30.608855 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.608857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.608883 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:57:30.608934 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.608935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.608943 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.608996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.609568 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.610031 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.610697 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.610868 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.610916 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.612009 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.612010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.612010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:30.612016 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.613160 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.613519 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.613520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.613521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.613570 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:30.613718 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:30.613719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:30.613719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:30.613842 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.613909 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.614005 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.614061 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:57:30.615854 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:30.617455 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:30.624038 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.619730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:36925 (local address 127.0.0.1:51308)
0925 14:57:30.620019 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.620071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.620138 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.620483 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.620484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.620485 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:30.620522 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.620524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.621913 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.621914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.622751 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.622753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.623730 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.623807 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.623823 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.623897 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.623947 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":58,"threads_started":1}
I20260925 14:57:30.624222 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.620000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.20.199.2:39803)
0925 14:57:30.620204 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.620209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.620323 (+ 114us) server_negotiation.cc:407] Connection header received
0925 14:57:30.620368 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.620370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.620384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.620527 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:30.620762 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.621552 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623017 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.623696 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623754 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.623798 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.624084 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.624128 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260925 14:57:30.624387 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.619824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.20.199.1:36925)
0925 14:57:30.620295 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.620299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.620310 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:30.620353 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.620355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.620371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.620601 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:30.620678 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.621800 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.622840 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.623653 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623733 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.623795 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.624253 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.624304 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":96,"threads_started":1}
I20260925 14:57:30.624482 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.620017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.20.199.3:37157)
0925 14:57:30.620212 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.620217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.620226 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:30.620263 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.620265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.620282 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.620358 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:30.620544 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.621761 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623245 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.623830 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.623957 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.624070 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.624387 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260925 14:57:30.624543 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.619735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:37157 (local address 127.0.0.1:38704)
0925 14:57:30.620037 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.620074 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.620138 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.620383 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.620386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.620388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:30.620460 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.620462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.621863 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.621864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.623169 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.623170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.623924 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.623993 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.623995 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.624055 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.624091 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
I20260925 14:57:30.624718 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.619750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:39803 (local address 127.0.0.1:51328)
0925 14:57:30.620241 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.620250 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.620305 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.620605 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.620605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.620607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:30.620662 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.620664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.621647 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.621648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.622936 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.622938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.623833 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.623834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.623899 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.623904 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.623957 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.624015 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260925 14:57:30.625100 8104 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9faa1ab75f473fa4b65aabbd227dad (DEFAULT_TABLE table=test-table [id=d18faa9c74d14e0284a8ef28b262299a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:30.625168 8373 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9faa1ab75f473fa4b65aabbd227dad (DEFAULT_TABLE table=test-table [id=d18faa9c74d14e0284a8ef28b262299a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:30.625511 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9faa1ab75f473fa4b65aabbd227dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:30.625511 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9faa1ab75f473fa4b65aabbd227dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:30.625435 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9faa1ab75f473fa4b65aabbd227dad (DEFAULT_TABLE table=test-table [id=d18faa9c74d14e0284a8ef28b262299a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:30.625797 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9faa1ab75f473fa4b65aabbd227dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:30.628595 8456 tablet_bootstrap.cc:492] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Bootstrap starting.
I20260925 14:57:30.629276 8456 tablet_bootstrap.cc:654] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:30.629575 8457 tablet_bootstrap.cc:492] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Bootstrap starting.
I20260925 14:57:30.629590 8458 tablet_bootstrap.cc:492] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: Bootstrap starting.
I20260925 14:57:30.629900 8456 log.cc:824] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:30.630236 8457 tablet_bootstrap.cc:654] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:30.630247 8458 tablet_bootstrap.cc:654] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:30.630723 8457 log.cc:824] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:30.630766 8456 tablet_bootstrap.cc:492] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: No bootstrap required, opened a new log
I20260925 14:57:30.630829 8456 ts_tablet_manager.cc:1397] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:57:30.630813 8458 log.cc:824] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:30.631494 8457 tablet_bootstrap.cc:492] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: No bootstrap required, opened a new log
I20260925 14:57:30.631520 8458 tablet_bootstrap.cc:492] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: No bootstrap required, opened a new log
I20260925 14:57:30.631575 8457 ts_tablet_manager.cc:1397] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:30.631604 8458 ts_tablet_manager.cc:1397] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:30.632854 8456 raft_consensus.cc:348] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.632930 8458 raft_consensus.cc:348] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.633054 8456 raft_consensus.cc:374] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:30.633080 8458 raft_consensus.cc:374] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:30.633092 8456 raft_consensus.cc:729] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fececfec014cd493e9e4fd8742884f, State: Initialized, Role: FOLLOWER
I20260925 14:57:30.633126 8458 raft_consensus.cc:729] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea9b36558a7477daddb8b55a7a6d163, State: Initialized, Role: FOLLOWER
I20260925 14:57:30.633237 8456 consensus_queue.cc:260] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.633252 8458 consensus_queue.cc:260] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.633549 8456 ts_tablet_manager.cc:1428] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:57:30.633569 8458 ts_tablet_manager.cc:1428] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:30.633550 8457 raft_consensus.cc:348] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.633724 8457 raft_consensus.cc:374] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:30.633769 8171 heartbeater.cc:503] Master 127.20.199.62:37903 was elected leader, sending a full tablet report...
I20260925 14:57:30.633777 8457 raft_consensus.cc:729] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb1374e57854db9887e3a920cddb46d, State: Initialized, Role: FOLLOWER
I20260925 14:57:30.633792 8305 heartbeater.cc:503] Master 127.20.199.62:37903 was elected leader, sending a full tablet report...
I20260925 14:57:30.633879 8457 consensus_queue.cc:260] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.634133 8457 ts_tablet_manager.cc:1428] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:30.634155 8439 heartbeater.cc:503] Master 127.20.199.62:37903 was elected leader, sending a full tablet report...
W20260925 14:57:30.677907 8306 tablet.cc:2369] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:30.729554 8464 raft_consensus.cc:482] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:30.729727 8464 raft_consensus.cc:504] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.730130 8464 leader_election.cc:288] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea9b36558a7477daddb8b55a7a6d163 (127.20.199.1:36925), 50fececfec014cd493e9e4fd8742884f (127.20.199.2:39803)
I20260925 14:57:30.733119 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.730431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:54627 (local address 127.20.199.1:36925)
0925 14:57:30.730472 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.730474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.730658 (+ 184us) server_negotiation.cc:407] Connection header received
0925 14:57:30.730703 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.730704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.730719 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.730772 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:30.730952 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.731428 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.732391 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.732763 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.732850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.732859 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.733001 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.733045 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:30.733143 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.730422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:51333 (local address 127.20.199.2:39803)
0925 14:57:30.730469 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.730472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.730484 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:57:30.730575 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.730577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.730590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.730648 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:30.730822 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.731368 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.732392 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.732765 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.732852 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.732861 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.733042 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.733083 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:30.733125 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.730342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:36925 (local address 127.20.199.3:54627)
0925 14:57:30.730558 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.730569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.730639 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.730800 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.730801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.730803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:30.730874 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.730876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.731528 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.731528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.732290 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.732293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.732846 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.732847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.732925 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.732929 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.732972 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.733025 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260925 14:57:30.733127 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.730333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:39803 (local address 127.20.199.3:51333)
0925 14:57:30.730425 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.730439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:30.730490 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:30.730669 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:30.730670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:30.730672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:30.730749 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.730752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.731452 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.731453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.732294 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:30.732296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.732853 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.732853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:30.732941 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.732944 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:30.733018 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:57:30.733053 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":48}
I20260925 14:57:30.733683 8125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9faa1ab75f473fa4b65aabbd227dad" candidate_uuid: "cbb1374e57854db9887e3a920cddb46d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea9b36558a7477daddb8b55a7a6d163" is_pre_election: true
I20260925 14:57:30.733767 8259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9faa1ab75f473fa4b65aabbd227dad" candidate_uuid: "cbb1374e57854db9887e3a920cddb46d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fececfec014cd493e9e4fd8742884f" is_pre_election: true
I20260925 14:57:30.733891 8259 raft_consensus.cc:2454] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbb1374e57854db9887e3a920cddb46d in term 0.
I20260925 14:57:30.733891 8125 raft_consensus.cc:2454] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbb1374e57854db9887e3a920cddb46d in term 0.
I20260925 14:57:30.734154 8326 leader_election.cc:302] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aea9b36558a7477daddb8b55a7a6d163, cbb1374e57854db9887e3a920cddb46d; no voters:
I20260925 14:57:30.734290 8464 raft_consensus.cc:2781] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:30.734345 8464 raft_consensus.cc:482] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:30.734369 8464 raft_consensus.cc:3037] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:30.735056 8464 raft_consensus.cc:504] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.735204 8464 leader_election.cc:288] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [CANDIDATE]: Term 1 election: Requested vote from peers aea9b36558a7477daddb8b55a7a6d163 (127.20.199.1:36925), 50fececfec014cd493e9e4fd8742884f (127.20.199.2:39803)
I20260925 14:57:30.735350 8125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9faa1ab75f473fa4b65aabbd227dad" candidate_uuid: "cbb1374e57854db9887e3a920cddb46d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea9b36558a7477daddb8b55a7a6d163"
I20260925 14:57:30.735360 8259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9faa1ab75f473fa4b65aabbd227dad" candidate_uuid: "cbb1374e57854db9887e3a920cddb46d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fececfec014cd493e9e4fd8742884f"
I20260925 14:57:30.735437 8125 raft_consensus.cc:3037] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:30.735437 8259 raft_consensus.cc:3037] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:30.736310 8259 raft_consensus.cc:2454] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb1374e57854db9887e3a920cddb46d in term 1.
I20260925 14:57:30.736310 8125 raft_consensus.cc:2454] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb1374e57854db9887e3a920cddb46d in term 1.
I20260925 14:57:30.736514 8328 leader_election.cc:302] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fececfec014cd493e9e4fd8742884f, cbb1374e57854db9887e3a920cddb46d; no voters:
I20260925 14:57:30.736649 8464 raft_consensus.cc:2781] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:30.736846 8464 raft_consensus.cc:686] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 1 LEADER]: Becoming Leader. State: Replica: cbb1374e57854db9887e3a920cddb46d, State: Running, Role: LEADER
I20260925 14:57:30.736971 8464 consensus_queue.cc:237] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } }
I20260925 14:57:30.737905 7978 catalog_manager.cc:5495] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d reported cstate change: term changed from 0 to 1, leader changed from <none> to cbb1374e57854db9887e3a920cddb46d (127.20.199.3). New cstate: current_term: 1 leader_uuid: "cbb1374e57854db9887e3a920cddb46d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb1374e57854db9887e3a920cddb46d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 } health_report { overall_health: UNKNOWN } } }
W20260925 14:57:30.754961 8172 tablet.cc:2369] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:30.771572 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.768862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.20.199.3:37157)
0925 14:57:30.768925 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.768928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.768937 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:30.768986 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.768986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.769005 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.769088 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:30.769234 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.769790 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.770680 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.770875 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.770948 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.771133 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:30.771269 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:30.771351 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.771401 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.771418 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:57:30.773537 8125 raft_consensus.cc:1261] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 1 FOLLOWER]: Refusing update from remote peer cbb1374e57854db9887e3a920cddb46d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:30.773607 8259 raft_consensus.cc:1261] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 1 FOLLOWER]: Refusing update from remote peer cbb1374e57854db9887e3a920cddb46d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:30.773883 8464 consensus_queue.cc:1035] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea9b36558a7477daddb8b55a7a6d163" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 36925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:30.774035 8466 consensus_queue.cc:1035] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fececfec014cd493e9e4fd8742884f" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 39803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:30.776332 8469 mvcc.cc:205] Tried to move back new op lower bound from 7333266435165425664 to 7333266435019452416. Current Snapshot: MvccSnapshot[applied={T|T < 7333266435165425664}]
I20260925 14:57:30.776727 8470 mvcc.cc:205] Tried to move back new op lower bound from 7333266435165425664 to 7333266435019452416. Current Snapshot: MvccSnapshot[applied={T|T < 7333266435165425664}]
I20260925 14:57:30.777184 8471 mvcc.cc:205] Tried to move back new op lower bound from 7333266435165425664 to 7333266435019452416. Current Snapshot: MvccSnapshot[applied={T|T < 7333266435165425664}]
I20260925 14:57:30.779126 7979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:30.779217 7979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:30.779969 7979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5f17f74119543d4b39b7cb9e60e2df3: Sending DeleteTablet for 3 replicas of tablet 6c9faa1ab75f473fa4b65aabbd227dad
I20260925 14:57:30.780280 8373 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9faa1ab75f473fa4b65aabbd227dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:30 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:38704
I20260925 14:57:30.780323 8104 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9faa1ab75f473fa4b65aabbd227dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:30 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:51308
I20260925 14:57:30.780369 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9faa1ab75f473fa4b65aabbd227dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:30 UTC) from {username='kudu', principal='kudu/127.20.199.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260925 14:57:30.780532 8478 tablet_replica.cc:330] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: stopping tablet replica
I20260925 14:57:30.780543 8479 tablet_replica.cc:330] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163: stopping tablet replica
I20260925 14:57:30.780623 8478 raft_consensus.cc:2229] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:30.780632 8479 raft_consensus.cc:2229] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:30.780728 8478 raft_consensus.cc:2258] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:30.780732 8479 raft_consensus.cc:2258] T 6c9faa1ab75f473fa4b65aabbd227dad P aea9b36558a7477daddb8b55a7a6d163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:30.780979 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8042
I20260925 14:57:30.781140 8480 tablet_replica.cc:330] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: stopping tablet replica
I20260925 14:57:30.781257 8480 raft_consensus.cc:2229] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:30.781427 8480 raft_consensus.cc:2258] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:30.781649 8478 ts_tablet_manager.cc:1905] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:30.781908 8480 ts_tablet_manager.cc:1905] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:30.783249 8478 ts_tablet_manager.cc:1918] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:30.783378 8478 log.cc:1196] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/6c9faa1ab75f473fa4b65aabbd227dad
I20260925 14:57:30.783725 8478 ts_tablet_manager.cc:1939] T 6c9faa1ab75f473fa4b65aabbd227dad P cbb1374e57854db9887e3a920cddb46d: Deleting consensus metadata
I20260925 14:57:30.783804 8480 ts_tablet_manager.cc:1918] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:30.783878 8480 log.cc:1196] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/6c9faa1ab75f473fa4b65aabbd227dad
I20260925 14:57:30.784144 8480 ts_tablet_manager.cc:1939] T 6c9faa1ab75f473fa4b65aabbd227dad P 50fececfec014cd493e9e4fd8742884f: Deleting consensus metadata
I20260925 14:57:30.784307 7967 catalog_manager.cc:4841] TS cbb1374e57854db9887e3a920cddb46d (127.20.199.3:37157): tablet 6c9faa1ab75f473fa4b65aabbd227dad (table test-table [id=d18faa9c74d14e0284a8ef28b262299a]) successfully deleted
I20260925 14:57:30.784727 7966 catalog_manager.cc:4841] TS 50fececfec014cd493e9e4fd8742884f (127.20.199.2:39803): tablet 6c9faa1ab75f473fa4b65aabbd227dad (table test-table [id=d18faa9c74d14e0284a8ef28b262299a]) successfully deleted
W20260925 14:57:30.788344 7964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:36925 (error 108)
I20260925 14:57:30.788533 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8176
I20260925 14:57:30.789776 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.789633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:36925 (local address 127.0.0.1:51310)
0925 14:57:30.789697 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:30.789738 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:36925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 14:57:30.790102 7964 catalog_manager.cc:4584] TS aea9b36558a7477daddb8b55a7a6d163 (127.20.199.1:36925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c9faa1ab75f473fa4b65aabbd227dad: Network error: Client connection negotiation failed: client connection to 127.20.199.1:36925: connect: Connection refused (error 111)
I20260925 14:57:30.792879 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8310
I20260925 14:57:30.797650 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 7948
2026-09-25T14:57:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T14:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:30Z Disabled control of system clock
I20260925 14:57:30.825596 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42825
--webserver_interface=127.20.199.62
--webserver_port=0
--builtin_ntp_servers=127.20.199.20:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:30.905654 8485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:30.905875 8485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:30.905895 8485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:30.906970 8485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:30.907002 8485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:30.907015 8485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:30.907027 8485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:30.907039 8485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:30.908288 8485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:42825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:30.908483 8485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:30.908726 8485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:30.913645 8490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.913724 8485 server_base.cc:1034] running on GCE node
W20260925 14:57:30.913692 8491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:30.913681 8493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:30.914083 8485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:30.914450 8485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:30.915647 8485 hybrid_clock.cc:648] HybridClock initialized: now 1790348250915627 us; error 44 us; skew 500 ppm
I20260925 14:57:30.917026 8485 webserver.cc:460] Webserver started at http://127.20.199.62:45431/ using document root <none> and password file <none>
I20260925 14:57:30.917312 8485 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:30.917462 8485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:30.917645 8485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:30.918740 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "72cbd5ba5d4c4085b60302454f7c8b3e"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "b46085fbd4c47dfea38406927e8bfdb9"
server_key_iv: "113c2e4041b11cdf0d16882bc02cd92b"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.919123 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "72cbd5ba5d4c4085b60302454f7c8b3e"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "b46085fbd4c47dfea38406927e8bfdb9"
server_key_iv: "113c2e4041b11cdf0d16882bc02cd92b"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.920457 8485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:30.921201 8499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:30.921449 8485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:30.921566 8485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "72cbd5ba5d4c4085b60302454f7c8b3e"
format_stamp: "Formatted at 2026-09-25 14:57:30 on dist-test-slave-8qt6"
server_key: "b46085fbd4c47dfea38406927e8bfdb9"
server_key_iv: "113c2e4041b11cdf0d16882bc02cd92b"
server_key_version: "encryptionkey@0"
I20260925 14:57:30.921658 8485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:30.938874 8485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:30.939224 8485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:30.939388 8485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:30.943391 8485 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:42825
I20260925 14:57:30.943430 8551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:42825 every 8 connection(s)
I20260925 14:57:30.943797 8485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:30.944815 8552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:30.947094 8552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e: Bootstrap starting.
I20260925 14:57:30.947629 8552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:30.947862 8552 log.cc:824] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:30.948489 8552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e: No bootstrap required, opened a new log
I20260925 14:57:30.950201 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8485
I20260925 14:57:30.950158 8552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42825 } }
I20260925 14:57:30.950312 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:30.950284 8552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:30.950356 8552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cbd5ba5d4c4085b60302454f7c8b3e, State: Initialized, Role: FOLLOWER
I20260925 14:57:30.950542 21276 external_mini_cluster.cc:1428] Setting key 9e4aafd1feee57d489ae2cb854a1d793
I20260925 14:57:30.950551 8552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42825 } }
I20260925 14:57:30.950659 8552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:30.950696 8552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:30.950748 8552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:30.951448 8552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42825 } }
I20260925 14:57:30.951568 8552 leader_election.cc:302] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72cbd5ba5d4c4085b60302454f7c8b3e; no voters:
I20260925 14:57:30.951747 8552 leader_election.cc:288] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:30.951794 8557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:30.951974 8557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [term 1 LEADER]: Becoming Leader. State: Replica: 72cbd5ba5d4c4085b60302454f7c8b3e, State: Running, Role: LEADER
I20260925 14:57:30.952005 8552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:30.952076 8557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42825 } }
I20260925 14:57:30.952378 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72cbd5ba5d4c4085b60302454f7c8b3e. Latest consensus state: current_term: 1 leader_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42825 } } }
I20260925 14:57:30.952476 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:30.952390 8559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbd5ba5d4c4085b60302454f7c8b3e" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42825 } } }
I20260925 14:57:30.952559 8559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:30.953338 8563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:30.953841 8563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:30.955848 8563 catalog_manager.cc:1347] Generated new cluster ID: f14046992b274125af8beac6a55f9042
I20260925 14:57:30.955900 8563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:30.956993 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:30.951074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.20.199.62:42825)
0925 14:57:30.951296 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:30.951304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:30.951330 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:57:30.951421 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:30.951426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:30.951494 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:30.951646 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:30.951892 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.953769 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.954844 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:30.955280 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:30.955370 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:30.956145 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:30.956150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:30.956150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:30.956361 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:30.956409 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:30.956426 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:30.956509 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:30.956648 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 14:57:30.956734 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260925 14:57:30.964349 8563 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:30.964846 8563 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:30.973839 8563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e: Generated new TSK 0
I20260925 14:57:30.974056 8563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:30.975715 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.1:0
--local_ip_for_outbound_sockets=127.20.199.1
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42825
--builtin_ntp_servers=127.20.199.20:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:57:31.064724 8576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:31.064931 8576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:31.064953 8576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:31.066169 8576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:31.066210 8576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:31.066234 8576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.1
I20260925 14:57:31.067548 8576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.199.1
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.1
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:31.067785 8576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:31.068112 8576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:31.074157 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:31.074112 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:31.074132 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.074299 8576 server_base.cc:1034] running on GCE node
I20260925 14:57:31.074679 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:31.074916 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:31.076062 8576 hybrid_clock.cc:648] HybridClock initialized: now 1790348251076041 us; error 38 us; skew 500 ppm
I20260925 14:57:31.077358 8576 webserver.cc:460] Webserver started at http://127.20.199.1:41391/ using document root <none> and password file <none>
I20260925 14:57:31.077688 8576 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:31.077739 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:31.077868 8576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:31.078941 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data/instance:
uuid: "b20aa57bf75f4e65b352949a029178af"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "1b040a339fb50feea58df9c01fb794f4"
server_key_iv: "80c7d2b09aa0cb1b2e789df8d0e93244"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.079309 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance:
uuid: "b20aa57bf75f4e65b352949a029178af"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "1b040a339fb50feea58df9c01fb794f4"
server_key_iv: "80c7d2b09aa0cb1b2e789df8d0e93244"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.080652 8576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:57:31.081513 8590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.081707 8576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:31.081800 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal
uuid: "b20aa57bf75f4e65b352949a029178af"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "1b040a339fb50feea58df9c01fb794f4"
server_key_iv: "80c7d2b09aa0cb1b2e789df8d0e93244"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.081897 8576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:31.098685 8576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:31.099025 8576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:31.099179 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:31.099431 8576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:31.099776 8576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:31.099833 8576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.099875 8576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:31.099901 8576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.105532 8576 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.1:35387
I20260925 14:57:31.105634 8703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.1:35387 every 8 connection(s)
I20260925 14:57:31.105948 8576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:31.110714 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.106694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:59983 (local address 127.20.199.62:42825)
0925 14:57:31.106777 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.106779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.106982 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:57:31.107431 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.107432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.107444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.107501 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:31.108134 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.108611 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.109546 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.109729 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.109782 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.110505 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:31.110506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:31.110506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:31.110510 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:31.110524 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:31.110528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:31.110570 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.110631 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.110656 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:57:31.110888 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.106659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42825 (local address 127.20.199.1:59983)
0925 14:57:31.106853 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.106887 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.107351 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.107559 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.107562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.107569 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:31.108066 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.108070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.108688 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.108689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.109463 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.109464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.109804 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.109805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.109907 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.110378 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:31.110378 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:31.110406 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:31.110427 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:31.110570 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:31.110572 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:31.110575 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.110621 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.110693 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260925 14:57:31.111465 8704 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42825
I20260925 14:57:31.111479 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8576
I20260925 14:57:31.111570 8704 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:31.111593 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:31.111771 8704 heartbeater.cc:511] Master 127.20.199.62:42825 requested a full tablet report, sending...
I20260925 14:57:31.111791 21276 external_mini_cluster.cc:1428] Setting key 312e2019b59f25c48fa7d3ea359dbede
I20260925 14:57:31.112272 8516 ts_manager.cc:194] Registered new tserver with Master: b20aa57bf75f4e65b352949a029178af (127.20.199.1:35387)
I20260925 14:57:31.112860 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.2:0
--local_ip_for_outbound_sockets=127.20.199.2
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42825
--builtin_ntp_servers=127.20.199.20:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:31.113003 8516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.1:59983
W20260925 14:57:31.194482 8707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:31.194681 8707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:31.194702 8707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:31.195860 8707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:31.195901 8707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:31.195953 8707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.2
I20260925 14:57:31.197365 8707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.199.2
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.2
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:31.197644 8707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:31.197938 8707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:31.202769 8715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:31.202860 8713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.202908 8707 server_base.cc:1034] running on GCE node
W20260925 14:57:31.202824 8712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.203274 8707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:31.203467 8707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:31.204604 8707 hybrid_clock.cc:648] HybridClock initialized: now 1790348251204574 us; error 50 us; skew 500 ppm
I20260925 14:57:31.205788 8707 webserver.cc:460] Webserver started at http://127.20.199.2:40233/ using document root <none> and password file <none>
I20260925 14:57:31.206054 8707 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:31.206101 8707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:31.206190 8707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:31.207185 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data/instance:
uuid: "83b3e8094d174c19a9ff29717fd121fe"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "d69ffcd6f8570ef22ef049eefb914d77"
server_key_iv: "b5a80249de602b7474628ff60fa98003"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.207466 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance:
uuid: "83b3e8094d174c19a9ff29717fd121fe"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "d69ffcd6f8570ef22ef049eefb914d77"
server_key_iv: "b5a80249de602b7474628ff60fa98003"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.208645 8707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:31.209301 8721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.209481 8707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:57:31.209571 8707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal
uuid: "83b3e8094d174c19a9ff29717fd121fe"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "d69ffcd6f8570ef22ef049eefb914d77"
server_key_iv: "b5a80249de602b7474628ff60fa98003"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.209625 8707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:31.239022 8707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:31.239302 8707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:31.239408 8707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:31.239590 8707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:31.239948 8707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:31.239995 8707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.240017 8707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:31.240029 8707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.245811 8707 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.2:34339
I20260925 14:57:31.245890 8834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.2:34339 every 8 connection(s)
I20260925 14:57:31.246218 8707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:31.247982 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8707
I20260925 14:57:31.248106 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:31.248292 21276 external_mini_cluster.cc:1428] Setting key fcb5d6fcd27d24d804da63c4d1bb675d
I20260925 14:57:31.249334 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.199.3:0
--local_ip_for_outbound_sockets=127.20.199.3
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42825
--builtin_ntp_servers=127.20.199.20:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:57:31.251823 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.247042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.2:60905 (local address 127.20.199.62:42825)
0925 14:57:31.247114 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.247117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.247315 (+ 198us) server_negotiation.cc:407] Connection header received
0925 14:57:31.247854 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.247855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.247868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.247932 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:31.248800 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.249364 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.250406 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.250620 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.250708 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.251576 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:31.251578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:31.251578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:31.251583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:31.251599 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:31.251603 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:31.251651 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.251719 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.251748 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:57:31.252053 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.247003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42825 (local address 127.20.199.2:60905)
0925 14:57:31.247186 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.247227 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.247741 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.248011 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.248015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.248025 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:31.248695 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.248702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.249502 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.249503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.250312 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.250314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.250718 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.250720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.250862 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.251423 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:31.251424 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:31.251453 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:31.251477 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:31.251650 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:31.251656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:31.251659 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.251725 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.251823 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260925 14:57:31.252521 8835 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42825
I20260925 14:57:31.252614 8835 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:31.252791 8835 heartbeater.cc:511] Master 127.20.199.62:42825 requested a full tablet report, sending...
I20260925 14:57:31.253180 8516 ts_manager.cc:194] Registered new tserver with Master: 83b3e8094d174c19a9ff29717fd121fe (127.20.199.2:34339)
I20260925 14:57:31.253613 8516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.2:60905
W20260925 14:57:31.329454 8838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:31.329667 8838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:31.329689 8838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:31.330761 8838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:31.330793 8838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:31.330817 8838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.199.3
I20260925 14:57:31.332039 8838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.199.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.199.3
--webserver_port=0
--tserver_master_addrs=127.20.199.62:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.199.3
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:31.332208 8838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:31.332434 8838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:31.337370 8843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:31.337446 8844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.337596 8838 server_base.cc:1034] running on GCE node
W20260925 14:57:31.337378 8846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.337863 8838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:31.338061 8838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:31.339289 8838 hybrid_clock.cc:648] HybridClock initialized: now 1790348251339275 us; error 31 us; skew 500 ppm
I20260925 14:57:31.340507 8838 webserver.cc:460] Webserver started at http://127.20.199.3:44729/ using document root <none> and password file <none>
I20260925 14:57:31.340716 8838 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:31.340757 8838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:31.340845 8838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:31.341936 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data/instance:
uuid: "6920ada5774346d8ba1f49cf94b7c31d"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "be98701d755f76f697e086a76020550a"
server_key_iv: "cab6f57006ec372a1d426d3972647b8c"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.342232 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance:
uuid: "6920ada5774346d8ba1f49cf94b7c31d"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "be98701d755f76f697e086a76020550a"
server_key_iv: "cab6f57006ec372a1d426d3972647b8c"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.343462 8838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:31.344260 8852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.344449 8838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:31.344520 8838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal
uuid: "6920ada5774346d8ba1f49cf94b7c31d"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-8qt6"
server_key: "be98701d755f76f697e086a76020550a"
server_key_iv: "cab6f57006ec372a1d426d3972647b8c"
server_key_version: "encryptionkey@0"
I20260925 14:57:31.344606 8838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:31.361145 8838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:31.361539 8838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:31.361691 8838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:31.361943 8838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:31.362272 8838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:31.362304 8838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.362345 8838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:31.362373 8838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:31.368084 8838 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.3:38663
I20260925 14:57:31.368134 8965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.3:38663 every 8 connection(s)
I20260925 14:57:31.368453 8838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:31.373304 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.369211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.3:36159 (local address 127.20.199.62:42825)
0925 14:57:31.369259 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.369261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.369584 (+ 323us) server_negotiation.cc:407] Connection header received
0925 14:57:31.370054 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.370056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.370066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.370131 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:31.370745 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.371229 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.372142 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.372317 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.372367 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.373079 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:31.373080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:31.373080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:31.373084 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:31.373096 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:31.373100 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:31.373142 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.373222 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.373246 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:31.373530 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.369127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:42825 (local address 127.20.199.3:36159)
0925 14:57:31.369384 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.369462 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.369946 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.370184 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.370187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.370195 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:31.370672 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.370676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.371307 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.371307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.372069 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.372070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.372385 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.372385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.372489 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.372954 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:57:31.372955 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:31.372982 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:57:31.373004 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:31.373155 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:31.373158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:31.373160 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.373210 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.373283 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260925 14:57:31.374131 8966 heartbeater.cc:347] Connected to a master server at 127.20.199.62:42825
I20260925 14:57:31.374220 8966 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:31.374364 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 8838
I20260925 14:57:31.374402 8966 heartbeater.cc:511] Master 127.20.199.62:42825 requested a full tablet report, sending...
I20260925 14:57:31.374487 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:31.374707 21276 external_mini_cluster.cc:1428] Setting key 94b25a375f755cdcbdcaac8d4a0a7f20
I20260925 14:57:31.374794 8516 ts_manager.cc:194] Registered new tserver with Master: 6920ada5774346d8ba1f49cf94b7c31d (127.20.199.3:38663)
I20260925 14:57:31.375289 8516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.199.3:36159
I20260925 14:57:31.375702 21276 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:31.379904 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.376977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.20.199.62:42825)
0925 14:57:31.377036 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.377038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.377145 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:57:31.377220 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.377221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.377234 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.377277 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:31.377482 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.377934 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.378707 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.378906 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.378981 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.379674 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:31.379675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:31.379675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:31.379679 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:31.379692 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:31.379696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:31.379738 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.379802 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.379824 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:57:31.381482 8515 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:31.382925 8515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:31.389586 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.385088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:35387 (local address 127.0.0.1:51638)
0925 14:57:31.385274 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.385312 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.385378 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.385898 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.385898 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.385901 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:31.385963 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.385974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.387277 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.387278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.388334 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.388337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.389246 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.389247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.389323 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.389336 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.389438 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.389484 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":48,"thread_start_us":56,"threads_started":1}
I20260925 14:57:31.389622 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.385170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.20.199.1:35387)
0925 14:57:31.385637 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.385640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.385648 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:31.385721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.385723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.385739 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.386031 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:31.386101 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.387192 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.388424 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.389156 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.389233 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.389329 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.389461 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.389500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":88,"threads_started":1}
I20260925 14:57:31.389729 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.385118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:34339 (local address 127.0.0.1:54418)
0925 14:57:31.385573 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.385581 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.385657 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.385892 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.385894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.385895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:31.385973 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.385975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.387255 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.387256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.388517 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.388520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.389456 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.389457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.389524 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.389527 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.389597 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.389638 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":33,"thread_start_us":86,"threads_started":1}
I20260925 14:57:31.389984 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.385358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.20.199.2:34339)
0925 14:57:31.385572 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.385576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.385650 (+ 74us) server_negotiation.cc:407] Connection header received
0925 14:57:31.385727 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.385730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.385746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.385819 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:31.386051 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.387166 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.388600 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.389273 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.389331 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.389421 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.389840 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.389887 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260925 14:57:31.390556 8769 tablet_service.cc:1467] Processing CreateTablet for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 (DEFAULT_TABLE table=test-table [id=e623367260b9454fa097453a060c14b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:31.390556 8638 tablet_service.cc:1467] Processing CreateTablet for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 (DEFAULT_TABLE table=test-table [id=e623367260b9454fa097453a060c14b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:31.390916 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:31.390916 8638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:31.394340 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.385118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:38663 (local address 127.0.0.1:33842)
0925 14:57:31.387427 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.387438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.387494 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.387727 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.387728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.387730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:31.387768 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.387769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.388879 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.388880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.390026 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.390028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.394092 (+ 4064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.394093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.394136 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.394139 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.394217 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.394257 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2296,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260925 14:57:31.394469 8981 tablet_bootstrap.cc:492] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Bootstrap starting.
I20260925 14:57:31.394647 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.385333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.20.199.3:38663)
0925 14:57:31.385519 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.385530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.387523 (+ 1993us) server_negotiation.cc:407] Connection header received
0925 14:57:31.387568 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.387570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.387586 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.387664 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:31.387839 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.388802 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.393491 (+ 4689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.393954 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.394005 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.394059 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.394232 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.394550 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260925 14:57:31.395223 8981 tablet_bootstrap.cc:654] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:31.395332 8900 tablet_service.cc:1467] Processing CreateTablet for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 (DEFAULT_TABLE table=test-table [id=e623367260b9454fa097453a060c14b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:31.395560 8981 log.cc:824] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:31.395680 8900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:31.395885 8980 tablet_bootstrap.cc:492] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: Bootstrap starting.
I20260925 14:57:31.396515 8980 tablet_bootstrap.cc:654] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:31.396808 8980 log.cc:824] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:31.397578 8981 tablet_bootstrap.cc:492] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: No bootstrap required, opened a new log
I20260925 14:57:31.397588 8980 tablet_bootstrap.cc:492] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: No bootstrap required, opened a new log
I20260925 14:57:31.397653 8981 ts_tablet_manager.cc:1397] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:57:31.397653 8980 ts_tablet_manager.cc:1397] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:31.398696 8984 tablet_bootstrap.cc:492] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Bootstrap starting.
I20260925 14:57:31.399287 8984 tablet_bootstrap.cc:654] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:31.399590 8984 log.cc:824] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:31.400214 8984 tablet_bootstrap.cc:492] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: No bootstrap required, opened a new log
I20260925 14:57:31.400291 8984 ts_tablet_manager.cc:1397] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:31.400260 8981 raft_consensus.cc:348] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.400260 8980 raft_consensus.cc:348] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.400429 8980 raft_consensus.cc:374] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:31.400484 8980 raft_consensus.cc:729] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20aa57bf75f4e65b352949a029178af, State: Initialized, Role: FOLLOWER
I20260925 14:57:31.400429 8981 raft_consensus.cc:374] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:31.400650 8981 raft_consensus.cc:729] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b3e8094d174c19a9ff29717fd121fe, State: Initialized, Role: FOLLOWER
I20260925 14:57:31.400631 8980 consensus_queue.cc:260] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.400777 8981 consensus_queue.cc:260] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.400882 8980 ts_tablet_manager.cc:1428] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:31.401013 8704 heartbeater.cc:503] Master 127.20.199.62:42825 was elected leader, sending a full tablet report...
I20260925 14:57:31.401024 8981 ts_tablet_manager.cc:1428] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:31.401162 8835 heartbeater.cc:503] Master 127.20.199.62:42825 was elected leader, sending a full tablet report...
I20260925 14:57:31.402890 8984 raft_consensus.cc:348] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.403015 8984 raft_consensus.cc:374] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:31.403036 8984 raft_consensus.cc:729] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6920ada5774346d8ba1f49cf94b7c31d, State: Initialized, Role: FOLLOWER
I20260925 14:57:31.403143 8984 consensus_queue.cc:260] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.403344 8984 ts_tablet_manager.cc:1428] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:31.403427 8966 heartbeater.cc:503] Master 127.20.199.62:42825 was elected leader, sending a full tablet report...
I20260925 14:57:31.473945 8986 raft_consensus.cc:482] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:31.474067 8986 raft_consensus.cc:504] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.474409 8986 leader_election.cc:288] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6920ada5774346d8ba1f49cf94b7c31d (127.20.199.3:38663), 83b3e8094d174c19a9ff29717fd121fe (127.20.199.2:34339)
I20260925 14:57:31.477547 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.474673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:43827 (local address 127.20.199.3:38663)
0925 14:57:31.474717 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.474719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.474729 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:31.474786 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.474787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.474795 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.474840 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:31.475032 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.475503 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.476716 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.477173 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.477256 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.477264 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.477438 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.477475 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:31.477550 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.474590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.3:38663 (local address 127.20.199.1:43827)
0925 14:57:31.474652 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.474665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.474717 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.474852 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.474853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.474854 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:31.474932 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.474934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.475588 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.475589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.476633 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.476636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.477253 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.477254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.477323 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.477326 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.477430 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.477480 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":34}
I20260925 14:57:31.477648 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.474657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.199.1:58713 (local address 127.20.199.2:34339)
0925 14:57:31.474705 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.474707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.474833 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:57:31.474922 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.474922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.474930 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.474992 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:31.475158 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.475608 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.476858 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.477232 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.477312 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.477320 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.477529 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.477575 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:31.477679 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.474602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.2:34339 (local address 127.20.199.1:58713)
0925 14:57:31.474765 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.474772 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:31.474848 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:31.474994 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:31.474995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:31.474997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:31.475081 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.475083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.475689 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.475690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.476762 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:31.476765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.477323 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.477324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:31.477455 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.477459 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:31.477504 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:57:31.477543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260925 14:57:31.477933 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ea4da5bf4f402eadfdc57a35e7b6e4" candidate_uuid: "b20aa57bf75f4e65b352949a029178af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b3e8094d174c19a9ff29717fd121fe" is_pre_election: true
I20260925 14:57:31.477937 8920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ea4da5bf4f402eadfdc57a35e7b6e4" candidate_uuid: "b20aa57bf75f4e65b352949a029178af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6920ada5774346d8ba1f49cf94b7c31d" is_pre_election: true
I20260925 14:57:31.478078 8789 raft_consensus.cc:2454] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20aa57bf75f4e65b352949a029178af in term 0.
I20260925 14:57:31.478094 8920 raft_consensus.cc:2454] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20aa57bf75f4e65b352949a029178af in term 0.
I20260925 14:57:31.478289 8591 leader_election.cc:302] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b3e8094d174c19a9ff29717fd121fe, b20aa57bf75f4e65b352949a029178af; no voters:
I20260925 14:57:31.478458 8986 raft_consensus.cc:2781] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:31.478523 8986 raft_consensus.cc:482] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:31.478541 8986 raft_consensus.cc:3037] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:31.479187 8986 raft_consensus.cc:504] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.479328 8986 leader_election.cc:288] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [CANDIDATE]: Term 1 election: Requested vote from peers 6920ada5774346d8ba1f49cf94b7c31d (127.20.199.3:38663), 83b3e8094d174c19a9ff29717fd121fe (127.20.199.2:34339)
I20260925 14:57:31.479450 8920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ea4da5bf4f402eadfdc57a35e7b6e4" candidate_uuid: "b20aa57bf75f4e65b352949a029178af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6920ada5774346d8ba1f49cf94b7c31d"
I20260925 14:57:31.479506 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ea4da5bf4f402eadfdc57a35e7b6e4" candidate_uuid: "b20aa57bf75f4e65b352949a029178af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b3e8094d174c19a9ff29717fd121fe"
I20260925 14:57:31.479528 8920 raft_consensus.cc:3037] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:31.479578 8789 raft_consensus.cc:3037] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:31.480204 8789 raft_consensus.cc:2454] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20aa57bf75f4e65b352949a029178af in term 1.
I20260925 14:57:31.480326 8920 raft_consensus.cc:2454] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20aa57bf75f4e65b352949a029178af in term 1.
I20260925 14:57:31.480381 8591 leader_election.cc:302] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b3e8094d174c19a9ff29717fd121fe, b20aa57bf75f4e65b352949a029178af; no voters:
I20260925 14:57:31.480458 8986 raft_consensus.cc:2781] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:31.480603 8986 raft_consensus.cc:686] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 1 LEADER]: Becoming Leader. State: Replica: b20aa57bf75f4e65b352949a029178af, State: Running, Role: LEADER
I20260925 14:57:31.480702 8986 consensus_queue.cc:237] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } }
I20260925 14:57:31.481554 8514 catalog_manager.cc:5495] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af reported cstate change: term changed from 0 to 1, leader changed from <none> to b20aa57bf75f4e65b352949a029178af (127.20.199.1). New cstate: current_term: 1 leader_uuid: "b20aa57bf75f4e65b352949a029178af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20aa57bf75f4e65b352949a029178af" member_type: VOTER last_known_addr { host: "127.20.199.1" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 } health_report { overall_health: UNKNOWN } } }
W20260925 14:57:31.496827 8836 tablet.cc:2369] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:31.504073 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.501606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.20.199.1:35387)
0925 14:57:31.501655 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:31.501657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:31.501670 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:57:31.501767 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:31.501768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:31.501797 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:31.501877 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:31.502038 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.502511 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.503263 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:31.503461 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:31.503543 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:31.503697 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:31.503802 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:31.503888 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:31.503940 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:57:31.503959 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:57:31.506213 8920 raft_consensus.cc:1261] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 1 FOLLOWER]: Refusing update from remote peer b20aa57bf75f4e65b352949a029178af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:31.506211 8789 raft_consensus.cc:1261] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 1 FOLLOWER]: Refusing update from remote peer b20aa57bf75f4e65b352949a029178af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:31.506642 8990 consensus_queue.cc:1035] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b3e8094d174c19a9ff29717fd121fe" member_type: VOTER last_known_addr { host: "127.20.199.2" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:31.506800 8986 consensus_queue.cc:1035] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6920ada5774346d8ba1f49cf94b7c31d" member_type: VOTER last_known_addr { host: "127.20.199.3" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:31.509387 8993 mvcc.cc:205] Tried to move back new op lower bound from 7333266438165692416 to 7333266438065483776. Current Snapshot: MvccSnapshot[applied={T|T < 7333266438165692416}]
I20260925 14:57:31.509686 8994 mvcc.cc:205] Tried to move back new op lower bound from 7333266438165692416 to 7333266438065483776. Current Snapshot: MvccSnapshot[applied={T|T < 7333266438165692416}]
I20260925 14:57:31.511884 8515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:31.511976 8515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:31.512718 8515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72cbd5ba5d4c4085b60302454f7c8b3e: Sending DeleteTablet for 3 replicas of tablet d7ea4da5bf4f402eadfdc57a35e7b6e4
I20260925 14:57:31.513064 8769 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:31 UTC) from {username='slave'} at 127.0.0.1:54418
I20260925 14:57:31.513073 8638 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:31 UTC) from {username='slave'} at 127.0.0.1:51638
I20260925 14:57:31.513088 8900 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:31 UTC) from {username='slave'} at 127.0.0.1:33842
I20260925 14:57:31.513296 9003 tablet_replica.cc:330] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af: stopping tablet replica
I20260925 14:57:31.513394 9003 raft_consensus.cc:2229] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P b20aa57bf75f4e65b352949a029178af [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:31.513543 9004 tablet_replica.cc:330] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: stopping tablet replica
I20260925 14:57:31.513636 9004 raft_consensus.cc:2229] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:31.513687 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8576
I20260925 14:57:31.513731 9004 raft_consensus.cc:2258] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:31.513440 9005 tablet_replica.cc:330] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: stopping tablet replica
I20260925 14:57:31.514221 9005 raft_consensus.cc:2229] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:31.514297 9005 raft_consensus.cc:2258] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:31.515424 9005 ts_tablet_manager.cc:1905] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:31.515439 9004 ts_tablet_manager.cc:1905] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:31.516889 9005 ts_tablet_manager.cc:1918] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:31.517014 9005 log.cc:1196] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-1/wal/wals/d7ea4da5bf4f402eadfdc57a35e7b6e4
I20260925 14:57:31.517261 9005 ts_tablet_manager.cc:1939] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 83b3e8094d174c19a9ff29717fd121fe: Deleting consensus metadata
I20260925 14:57:31.517338 9004 ts_tablet_manager.cc:1918] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:31.517433 9004 log.cc:1196] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Deleting WAL directory at /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348014035135-21276-0/minicluster-data/ts-2/wal/wals/d7ea4da5bf4f402eadfdc57a35e7b6e4
I20260925 14:57:31.517670 9004 ts_tablet_manager.cc:1939] T d7ea4da5bf4f402eadfdc57a35e7b6e4 P 6920ada5774346d8ba1f49cf94b7c31d: Deleting consensus metadata
I20260925 14:57:31.517742 8500 catalog_manager.cc:4841] TS 83b3e8094d174c19a9ff29717fd121fe (127.20.199.2:34339): tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 (table test-table [id=e623367260b9454fa097453a060c14b7]) successfully deleted
I20260925 14:57:31.518070 8501 catalog_manager.cc:4841] TS 6920ada5774346d8ba1f49cf94b7c31d (127.20.199.3:38663): tablet d7ea4da5bf4f402eadfdc57a35e7b6e4 (table test-table [id=e623367260b9454fa097453a060c14b7]) successfully deleted
W20260925 14:57:31.520056 8503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.199.1:35387 (error 108)
I20260925 14:57:31.520498 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:31.520361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.1:35387 (local address 127.0.0.1:51660)
0925 14:57:31.520416 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:31.520459 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.1:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:57:31.520540 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8707
W20260925 14:57:31.520639 8503 catalog_manager.cc:4584] TS b20aa57bf75f4e65b352949a029178af (127.20.199.1:35387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7ea4da5bf4f402eadfdc57a35e7b6e4: Network error: Client connection negotiation failed: client connection to 127.20.199.1:35387: connect: Connection refused (error 111)
I20260925 14:57:31.524838 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8838
I20260925 14:57:31.529502 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 8485
2026-09-25T14:57:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (730 ms)
[----------] 6 tests from EncryptionPolicyTest (8488 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:31 dist-test-slave-8qt6 krb5kdc[9009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:31 dist-test-slave-8qt6 krb5kdc[9009](info): set up 2 sockets
Sep 25 14:57:31 dist-test-slave-8qt6 krb5kdc[9009](info): commencing operation
krb5kdc: starting...
W20260925 14:57:32.573542 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:32 dist-test-slave-8qt6 krb5kdc[9009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:32.692899 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:32.774777 9022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:32.774977 9022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:32.775000 9022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:32.776142 9022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:32.776183 9022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:32.776198 9022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:32.776211 9022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:32.776263 9022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:32.777799 9022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:32.778043 9022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:32.778265 9022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:32.783233 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:32.783231 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:32.783226 9030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:32.783504 9022 server_base.cc:1034] running on GCE node
I20260925 14:57:32.783730 9022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:32.784049 9022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:32.785197 9022 hybrid_clock.cc:648] HybridClock initialized: now 1790348252785173 us; error 39 us; skew 500 ppm
Sep 25 14:57:32 dist-test-slave-8qt6 krb5kdc[9009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:32.787809 9022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:32.788286 9022 webserver.cc:460] Webserver started at http://127.0.0.1:33073/ using document root <none> and password file <none>
I20260925 14:57:32.788508 9022 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:32.788552 9022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:32.788731 9022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:32.789883 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "96a4a2c2e37d442fb4f54938eff1b5f9"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-8qt6"
server_key: "29588bf2044fc82d68972e8e182dd8d1"
server_key_iv: "7c39824f182aacc5cf413317c72285e6"
server_key_version: "encryptionkey@0"
I20260925 14:57:32.790246 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "96a4a2c2e37d442fb4f54938eff1b5f9"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-8qt6"
server_key: "29588bf2044fc82d68972e8e182dd8d1"
server_key_iv: "7c39824f182aacc5cf413317c72285e6"
server_key_version: "encryptionkey@0"
I20260925 14:57:32.791574 9022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:32.792325 9037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:32.792527 9022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:32.792632 9022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "96a4a2c2e37d442fb4f54938eff1b5f9"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-8qt6"
server_key: "29588bf2044fc82d68972e8e182dd8d1"
server_key_iv: "7c39824f182aacc5cf413317c72285e6"
server_key_version: "encryptionkey@0"
I20260925 14:57:32.792688 9022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:32.815140 9022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:32.817256 9022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:32.817399 9022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:32.821276 9022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39201
I20260925 14:57:32.821334 9089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39201 every 8 connection(s)
I20260925 14:57:32.821676 9022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:32.822567 9090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:32.824837 9090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9: Bootstrap starting.
I20260925 14:57:32.825441 9090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:32.825736 9090 log.cc:824] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:32.826408 9090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9: No bootstrap required, opened a new log
I20260925 14:57:32.827802 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9022
I20260925 14:57:32.827894 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:32.828102 21276 external_mini_cluster.cc:1428] Setting key 0372a1d82e65e20742bd04a43207f2fb
I20260925 14:57:32.828167 9090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260925 14:57:32.828372 9090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:32.828415 9090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a4a2c2e37d442fb4f54938eff1b5f9, State: Initialized, Role: FOLLOWER
I20260925 14:57:32.828804 9090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260925 14:57:32.828907 9090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:32.828939 9090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:32.828984 9090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:32.830083 9090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260925 14:57:32.830188 9090 leader_election.cc:302] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a4a2c2e37d442fb4f54938eff1b5f9; no voters:
I20260925 14:57:32.830349 9090 leader_election.cc:288] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:32.830416 9095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:32.830569 9095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 96a4a2c2e37d442fb4f54938eff1b5f9, State: Running, Role: LEADER
I20260925 14:57:32.830591 9090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:32.830662 9095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260925 14:57:32.830978 9095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a4a2c2e37d442fb4f54938eff1b5f9. Latest consensus state: current_term: 1 leader_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260925 14:57:32.831054 9095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:32.831210 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a4a2c2e37d442fb4f54938eff1b5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260925 14:57:32.831290 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:32.831403 9102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:32.832129 9102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:32.833988 9102 catalog_manager.cc:1347] Generated new cluster ID: a483e24a6e0b470fad406e33b5237ec6
I20260925 14:57:32.834044 9102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:32 dist-test-slave-8qt6 krb5kdc[9009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:32.839027 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.828652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47467 (local address 127.0.0.1:39201)
0925 14:57:32.828843 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:32.828852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:32.828885 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:57:32.829011 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:32.829019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:32.829076 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:32.829213 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:32.829899 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.831303 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.832512 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.833053 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.833160 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.835727 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:32.835730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:32.835731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:32.835743 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:32.837448 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.837856 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.837859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.837859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.837964 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.838194 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.838195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.838195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.838439 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:32.838632 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:32.838718 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:57:32.838823 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260925 14:57:32.840029 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:32.847779 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.841656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40655 (local address 127.0.0.1:39201)
0925 14:57:32.841692 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:32.841694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:32.841873 (+ 179us) server_negotiation.cc:407] Connection header received
0925 14:57:32.841906 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:32.841907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:32.841919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:32.841960 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:32.842463 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.842942 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.843670 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.843893 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.843977 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.845130 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:32.845131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:32.845131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:32.845138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:32.846588 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.846948 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.846949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.846950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.847014 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.847257 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.847259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.847259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.847372 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:32.847485 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:32.847601 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:57:32.847665 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:57:32.850172 9102 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:32.850903 9102 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:32.874606 9102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a4a2c2e37d442fb4f54938eff1b5f9: Generated new TSK 0
I20260925 14:57:32.874871 9102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:32.881866 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9022
W20260925 14:57:32.886678 9007 connection.cc:535] client connection to 127.0.0.1:39201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39201): Connection reset by peer (error 104)
2026-09-25T14:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:32 dist-test-slave-8qt6 krb5kdc[9121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:32 dist-test-slave-8qt6 krb5kdc[9121](info): set up 2 sockets
Sep 25 14:57:32 dist-test-slave-8qt6 krb5kdc[9121](info): commencing operation
krb5kdc: starting...
W20260925 14:57:33.933755 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:34 dist-test-slave-8qt6 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:34.053077 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:34.133414 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:34.133667 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:34.133705 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:34.134876 9134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:34.134949 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:34.134974 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:34.134997 9134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:34.135025 9134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:34.136493 9134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:34.136776 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:34.137010 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:34.141847 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:34.141876 9134 server_base.cc:1034] running on GCE node
W20260925 14:57:34.141956 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.141850 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:34.142307 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:34.142568 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:34.143714 9134 hybrid_clock.cc:648] HybridClock initialized: now 1790348254143700 us; error 30 us; skew 500 ppm
Sep 25 14:57:34 dist-test-slave-8qt6 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:34.146255 9134 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:34.146710 9134 webserver.cc:460] Webserver started at http://127.0.0.1:36569/ using document root <none> and password file <none>
I20260925 14:57:34.146898 9134 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:34.146936 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:34.147027 9134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:34.148036 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "6229f72dc31e4183a2db53e7877d8691"
format_stamp: "Formatted at 2026-09-25 14:57:34 on dist-test-slave-8qt6"
server_key: "0813b8b7753161fd6827068cad33f04d"
server_key_iv: "1439ccd6570d3be3cd4fe679cf2708c5"
server_key_version: "encryptionkey@0"
I20260925 14:57:34.148331 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "6229f72dc31e4183a2db53e7877d8691"
format_stamp: "Formatted at 2026-09-25 14:57:34 on dist-test-slave-8qt6"
server_key: "0813b8b7753161fd6827068cad33f04d"
server_key_iv: "1439ccd6570d3be3cd4fe679cf2708c5"
server_key_version: "encryptionkey@0"
I20260925 14:57:34.149672 9134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:57:34.150471 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.150671 9134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:34.150743 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "6229f72dc31e4183a2db53e7877d8691"
format_stamp: "Formatted at 2026-09-25 14:57:34 on dist-test-slave-8qt6"
server_key: "0813b8b7753161fd6827068cad33f04d"
server_key_iv: "1439ccd6570d3be3cd4fe679cf2708c5"
server_key_version: "encryptionkey@0"
I20260925 14:57:34.150799 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:34.166569 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:34.168955 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:34.169080 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:34.173825 9134 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46017
I20260925 14:57:34.173872 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46017 every 8 connection(s)
I20260925 14:57:34.174260 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:34.175326 9202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:34.177625 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691: Bootstrap starting.
I20260925 14:57:34.177937 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9134
I20260925 14:57:34.178082 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:34.178289 21276 external_mini_cluster.cc:1428] Setting key 2239929d5f1b4bd7420d2ca68719da67
I20260925 14:57:34.178398 9202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:34.178854 9202 log.cc:824] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:34.179788 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691: No bootstrap required, opened a new log
I20260925 14:57:34.182548 9202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6229f72dc31e4183a2db53e7877d8691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 14:57:34.182723 9202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:34.182770 9202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6229f72dc31e4183a2db53e7877d8691, State: Initialized, Role: FOLLOWER
I20260925 14:57:34.182912 9202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6229f72dc31e4183a2db53e7877d8691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 14:57:34.183012 9202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:34.183055 9202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:34.183104 9202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:34.183786 9202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6229f72dc31e4183a2db53e7877d8691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 14:57:34.183926 9202 leader_election.cc:302] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6229f72dc31e4183a2db53e7877d8691; no voters:
I20260925 14:57:34.184091 9202 leader_election.cc:288] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:34.184125 9207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:34.184296 9207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [term 1 LEADER]: Becoming Leader. State: Replica: 6229f72dc31e4183a2db53e7877d8691, State: Running, Role: LEADER
I20260925 14:57:34.184369 9202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:34.184384 9207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6229f72dc31e4183a2db53e7877d8691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 14:57:34.184707 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6229f72dc31e4183a2db53e7877d8691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6229f72dc31e4183a2db53e7877d8691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } } }
I20260925 14:57:34.184782 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:34.185122 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.178858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37893 (local address 127.0.0.1:46017)
0925 14:57:34.179133 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.179169 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.179271 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:57:34.180061 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.180072 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.180142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.180317 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.180565 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.181719 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.182829 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.183344 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.183432 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.184262 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.184266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.184267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:34.184480 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.184523 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:34.184570 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.184655 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.184838 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.184868 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260925 14:57:34.185105 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6229f72dc31e4183a2db53e7877d8691. Latest consensus state: current_term: 1 leader_uuid: "6229f72dc31e4183a2db53e7877d8691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6229f72dc31e4183a2db53e7877d8691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } } }
I20260925 14:57:34.185621 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:34.186224 9223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:34.186275 9223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:34.186364 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:34.186329 9212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:34.187813 9212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:34.189262 9212 catalog_manager.cc:1347] Generated new cluster ID: 884b3e4723bc455594d4906bc41cdf2c
I20260925 14:57:34.189314 9212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:34.196548 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.187826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52797 (local address 127.0.0.1:46017)
0925 14:57:34.187872 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.187874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.193613 (+ 5739us) server_negotiation.cc:407] Connection header received
0925 14:57:34.193707 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.193709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.193721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.193751 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.193997 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.194501 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.195231 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.195475 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.195561 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.196270 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.196271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.196272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:34.196277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.196292 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:34.196296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.196371 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.196433 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.196446 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:34.230577 9212 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:34.231132 9212 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:34.235872 9212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6229f72dc31e4183a2db53e7877d8691: Generated new TSK 0
I20260925 14:57:34.236063 9212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:34.239846 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9134
W20260925 14:57:34.244819 9119 connection.cc:535] client connection to 127.0.0.1:46017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46017): Connection reset by peer (error 104)
2026-09-25T14:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:34 dist-test-slave-8qt6 krb5kdc[9233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:34 dist-test-slave-8qt6 krb5kdc[9233](info): set up 2 sockets
Sep 25 14:57:34 dist-test-slave-8qt6 krb5kdc[9233](info): commencing operation
krb5kdc: starting...
W20260925 14:57:35.269373 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:35 dist-test-slave-8qt6 krb5kdc[9233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:35.388082 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:35.467818 9246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:35.468070 9246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:35.468112 9246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:35.469256 9246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:35.469324 9246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:35.469348 9246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:35.469369 9246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:35.469415 9246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:35.470903 9246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:35.471184 9246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:35.471405 9246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:35.476363 9246 server_base.cc:1034] running on GCE node
W20260925 14:57:35.476383 9251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:35.476334 9252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:35.476328 9254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:35.476822 9246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:35.477069 9246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:35.478245 9246 hybrid_clock.cc:648] HybridClock initialized: now 1790348255478225 us; error 39 us; skew 500 ppm
Sep 25 14:57:35 dist-test-slave-8qt6 krb5kdc[9233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:35.488652 9246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:35.489154 9246 webserver.cc:460] Webserver started at http://127.0.0.1:35331/ using document root <none> and password file <none>
I20260925 14:57:35.489436 9246 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:35.489507 9246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:35.489683 9246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:35.490819 9246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "4522005a63854e178685beb0b8320c5e"
format_stamp: "Formatted at 2026-09-25 14:57:35 on dist-test-slave-8qt6"
server_key: "5b302e0a5df8cf7320f89d2a451ae05a"
server_key_iv: "298df006f618edbf28446b298b3fc080"
server_key_version: "encryptionkey@0"
I20260925 14:57:35.491180 9246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "4522005a63854e178685beb0b8320c5e"
format_stamp: "Formatted at 2026-09-25 14:57:35 on dist-test-slave-8qt6"
server_key: "5b302e0a5df8cf7320f89d2a451ae05a"
server_key_iv: "298df006f618edbf28446b298b3fc080"
server_key_version: "encryptionkey@0"
I20260925 14:57:35.492594 9246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:35.493381 9261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:35.493577 9246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:35.493681 9246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "4522005a63854e178685beb0b8320c5e"
format_stamp: "Formatted at 2026-09-25 14:57:35 on dist-test-slave-8qt6"
server_key: "5b302e0a5df8cf7320f89d2a451ae05a"
server_key_iv: "298df006f618edbf28446b298b3fc080"
server_key_version: "encryptionkey@0"
I20260925 14:57:35.493772 9246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:35.497104 9246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:35.498987 9246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:35.499146 9246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:35.503172 9246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42829
I20260925 14:57:35.503234 9313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42829 every 8 connection(s)
I20260925 14:57:35.503566 9246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:35.504411 9314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:35.506880 9314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e: Bootstrap starting.
I20260925 14:57:35.507424 9314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:35.507654 9314 log.cc:824] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:35.508334 9314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e: No bootstrap required, opened a new log
I20260925 14:57:35.510066 9314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522005a63854e178685beb0b8320c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42829 } }
I20260925 14:57:35.510205 9314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:35.510227 9314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4522005a63854e178685beb0b8320c5e, State: Initialized, Role: FOLLOWER
I20260925 14:57:35.510363 9314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522005a63854e178685beb0b8320c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42829 } }
I20260925 14:57:35.510447 9314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:35.510469 9314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:35.510499 9314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:35.510990 9314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522005a63854e178685beb0b8320c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42829 } }
I20260925 14:57:35.511077 9314 leader_election.cc:302] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4522005a63854e178685beb0b8320c5e; no voters:
I20260925 14:57:35.511204 9314 leader_election.cc:288] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:35.511281 9317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:35.511445 9317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [term 1 LEADER]: Becoming Leader. State: Replica: 4522005a63854e178685beb0b8320c5e, State: Running, Role: LEADER
I20260925 14:57:35.511478 9314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:35.511571 9317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522005a63854e178685beb0b8320c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42829 } }
I20260925 14:57:35.512051 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4522005a63854e178685beb0b8320c5e. Latest consensus state: current_term: 1 leader_uuid: "4522005a63854e178685beb0b8320c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522005a63854e178685beb0b8320c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42829 } } }
I20260925 14:57:35.512143 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:35.512183 9318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4522005a63854e178685beb0b8320c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522005a63854e178685beb0b8320c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42829 } } }
I20260925 14:57:35.512290 9318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:35.512382 9322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:35.512607 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9246
I20260925 14:57:35.512723 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:35.512944 21276 external_mini_cluster.cc:1428] Setting key 711a042077d2e5590ad2b7006f30ca70
I20260925 14:57:35.512961 9322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:35.515486 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.513616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33327 (local address 127.0.0.1:42829)
0925 14:57:35.513968 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.513975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.513999 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:57:35.514154 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.514163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.514194 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.514359 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:35.515060 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:35.515063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:35.515063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:35.515067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:35.515117 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:35.515128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:35.515204 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.515275 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.515322 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":76,"threads_started":1}
I20260925 14:57:35.516525 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:35.517860 9322 catalog_manager.cc:1347] Generated new cluster ID: 5e8131a14c5a41f1bf160b79bbedcdb7
I20260925 14:57:35.517907 9322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:35.518945 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.517854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50561 (local address 127.0.0.1:42829)
0925 14:57:35.517890 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.517892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.517993 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:57:35.518066 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.518067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.518072 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.518120 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:35.518728 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:35.518729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:35.518729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:35.518733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:35.518747 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:35.518752 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:35.518809 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.518857 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.518872 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:35.538424 9322 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:35.538636 9322 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:35.548278 9322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4522005a63854e178685beb0b8320c5e: Generated new TSK 0
I20260925 14:57:35.548485 9322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:35.554513 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9246
2026-09-25T14:57:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:35 dist-test-slave-8qt6 krb5kdc[9345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:35 dist-test-slave-8qt6 krb5kdc[9345](info): set up 2 sockets
Sep 25 14:57:35 dist-test-slave-8qt6 krb5kdc[9345](info): commencing operation
krb5kdc: starting...
W20260925 14:57:36.593745 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:36 dist-test-slave-8qt6 krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:36.717509 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:36.798411 9358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:36.798658 9358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:36.798700 9358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:36.799855 9358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:36.799925 9358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:36.799949 9358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:36.799978 9358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:36.800007 9358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:36.801486 9358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:36.801786 9358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:36.802002 9358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:36.806731 9366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:36.806726 9363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:36.806757 9364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:36.807153 9358 server_base.cc:1034] running on GCE node
I20260925 14:57:36.807389 9358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:36.807641 9358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:36.808811 9358 hybrid_clock.cc:648] HybridClock initialized: now 1790348256808801 us; error 31 us; skew 500 ppm
Sep 25 14:57:36 dist-test-slave-8qt6 krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:36.811446 9358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:36.811946 9358 webserver.cc:460] Webserver started at http://127.0.0.1:44991/ using document root <none> and password file <none>
I20260925 14:57:36.812201 9358 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:36.812248 9358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:36.812394 9358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:36.813512 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "f3593eef9d4241b18aab16fe83697a39"
format_stamp: "Formatted at 2026-09-25 14:57:36 on dist-test-slave-8qt6"
server_key: "3205811f2637436865c6c1c884095c29"
server_key_iv: "a24e5e0aa855c05c196002c7192a83c7"
server_key_version: "encryptionkey@0"
I20260925 14:57:36.813884 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "f3593eef9d4241b18aab16fe83697a39"
format_stamp: "Formatted at 2026-09-25 14:57:36 on dist-test-slave-8qt6"
server_key: "3205811f2637436865c6c1c884095c29"
server_key_iv: "a24e5e0aa855c05c196002c7192a83c7"
server_key_version: "encryptionkey@0"
I20260925 14:57:36.815267 9358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:36.816135 9373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:36.816401 9358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:36.816502 9358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "f3593eef9d4241b18aab16fe83697a39"
format_stamp: "Formatted at 2026-09-25 14:57:36 on dist-test-slave-8qt6"
server_key: "3205811f2637436865c6c1c884095c29"
server_key_iv: "a24e5e0aa855c05c196002c7192a83c7"
server_key_version: "encryptionkey@0"
I20260925 14:57:36.816561 9358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:36.831867 9358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:36.833802 9358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:36.833917 9358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:36.837903 9358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38669
I20260925 14:57:36.837963 9425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38669 every 8 connection(s)
I20260925 14:57:36.838358 9358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:36.839296 9426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:36.841735 9426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39: Bootstrap starting.
I20260925 14:57:36.842394 9426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:36.842389 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9358
I20260925 14:57:36.842540 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:36.842728 21276 external_mini_cluster.cc:1428] Setting key 182fab350c1d69424fecebe2ae237603
I20260925 14:57:36.842751 9426 log.cc:824] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:36.843607 9426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39: No bootstrap required, opened a new log
I20260925 14:57:36.845760 9426 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3593eef9d4241b18aab16fe83697a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260925 14:57:36.845920 9426 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:36.845970 9426 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3593eef9d4241b18aab16fe83697a39, State: Initialized, Role: FOLLOWER
I20260925 14:57:36.846131 9426 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3593eef9d4241b18aab16fe83697a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260925 14:57:36.846225 9426 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:36.846271 9426 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:36.846318 9426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:36.846868 9426 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3593eef9d4241b18aab16fe83697a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260925 14:57:36.846987 9426 leader_election.cc:302] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3593eef9d4241b18aab16fe83697a39; no voters:
I20260925 14:57:36.847177 9426 leader_election.cc:288] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:36.847221 9431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:36.847378 9431 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [term 1 LEADER]: Becoming Leader. State: Replica: f3593eef9d4241b18aab16fe83697a39, State: Running, Role: LEADER
I20260925 14:57:36.847456 9426 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:36.847540 9431 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3593eef9d4241b18aab16fe83697a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260925 14:57:36.848029 9433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3593eef9d4241b18aab16fe83697a39. Latest consensus state: current_term: 1 leader_uuid: "f3593eef9d4241b18aab16fe83697a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3593eef9d4241b18aab16fe83697a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } } }
I20260925 14:57:36.848254 9433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:36.848188 9432 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3593eef9d4241b18aab16fe83697a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3593eef9d4241b18aab16fe83697a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } } }
I20260925 14:57:36.848502 9432 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:36.848642 9441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:36.849156 9441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:57:36 dist-test-slave-8qt6 krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:36.850819 9441 catalog_manager.cc:1347] Generated new cluster ID: 94830e471fe0463e9019a58748a4e8a2
I20260925 14:57:36.850878 9441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:36.854470 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:36.843307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41323 (local address 127.0.0.1:38669)
0925 14:57:36.843504 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:36.843513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:36.843551 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:57:36.844004 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:36.844011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:36.844067 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:36.844213 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:36.844865 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:36.845805 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:36.846728 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:36.847259 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:36.847410 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:36.850527 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:36.850531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:36.850531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:36.850544 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:36.852612 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:36.853037 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:36.853040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:36.853040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:36.853153 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:36.853465 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:36.853466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:36.853467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:36.853819 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:36.854002 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:36.854086 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:57:36.854241 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260925 14:57:36.855585 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:36.864972 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:36.857019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40431 (local address 127.0.0.1:38669)
0925 14:57:36.857056 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:36.857058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:36.857591 (+ 533us) server_negotiation.cc:407] Connection header received
0925 14:57:36.857654 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:36.857655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:36.857664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:36.857714 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:36.858398 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:36.859146 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:36.860220 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:36.860517 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:36.860575 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:36.862203 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:36.862204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:36.862205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:36.862211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:36.863801 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:36.864204 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:36.864205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:36.864205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:36.864262 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:36.864467 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:36.864467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:36.864467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:36.864590 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:36.864681 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:36.864804 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 14:57:36.864880 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:36.872246 9441 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:36.872789 9441 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:36.880206 9441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3593eef9d4241b18aab16fe83697a39: Generated new TSK 0
I20260925 14:57:36.880412 9441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:36.884809 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9358
2026-09-25T14:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:36 dist-test-slave-8qt6 krb5kdc[9457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:36 dist-test-slave-8qt6 krb5kdc[9457](info): set up 2 sockets
Sep 25 14:57:36 dist-test-slave-8qt6 krb5kdc[9457](info): commencing operation
krb5kdc: starting...
W20260925 14:57:37.917758 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:37 dist-test-slave-8qt6 krb5kdc[9457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:38.040340 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:38.121526 9470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:38.121796 9470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:38.121840 9470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:38.123054 9470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:38.123126 9470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:38.123150 9470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:38.123178 9470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:38.123210 9470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:38.124679 9470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:38.124953 9470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:38.125185 9470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:38.130338 9476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:38.130347 9478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:38.130386 9475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:38.130563 9470 server_base.cc:1034] running on GCE node
I20260925 14:57:38.130831 9470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:38.131093 9470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:38.132270 9470 hybrid_clock.cc:648] HybridClock initialized: now 1790348258132258 us; error 30 us; skew 500 ppm
Sep 25 14:57:38 dist-test-slave-8qt6 krb5kdc[9457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:38.134891 9470 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:38.135356 9470 webserver.cc:460] Webserver started at http://127.0.0.1:35573/ using document root <none> and password file <none>
I20260925 14:57:38.135560 9470 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:38.135601 9470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:38.135689 9470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:38.136741 9470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "01fb529a890e4dee845380c8cc066c8a"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-8qt6"
server_key: "89f9d7653a127731d6e6fbd69b184a6b"
server_key_iv: "d0c80903af034126663b8879aa48121e"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.137046 9470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "01fb529a890e4dee845380c8cc066c8a"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-8qt6"
server_key: "89f9d7653a127731d6e6fbd69b184a6b"
server_key_iv: "d0c80903af034126663b8879aa48121e"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.138391 9470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:38.139173 9485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:38.139380 9470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:38.139462 9470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "01fb529a890e4dee845380c8cc066c8a"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-8qt6"
server_key: "89f9d7653a127731d6e6fbd69b184a6b"
server_key_iv: "d0c80903af034126663b8879aa48121e"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.139531 9470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:38.147225 9470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:38.149340 9470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:38.149508 9470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:38.153164 9470 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38433
I20260925 14:57:38.153213 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38433 every 8 connection(s)
I20260925 14:57:38.153581 9470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:38.154422 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:38.154994 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9470
I20260925 14:57:38.155139 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:38.155330 21276 external_mini_cluster.cc:1428] Setting key a3d3fd4f10385d1bfcccd1fcb1326041
I20260925 14:57:38.157337 9538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a: Bootstrap starting.
I20260925 14:57:38.157951 9538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:38.158280 9538 log.cc:824] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:38.159000 9538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a: No bootstrap required, opened a new log
I20260925 14:57:38.160928 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.155964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60885 (local address 127.0.0.1:38433)
0925 14:57:38.156143 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.156150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.156173 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:57:38.156571 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.156577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.156622 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.156729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.156975 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.157922 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.158956 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.159440 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.159545 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.160234 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.160238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.160238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:38.160396 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.160426 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:38.160436 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.160495 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.160591 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.160699 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 14:57:38.161590 9538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fb529a890e4dee845380c8cc066c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38433 } }
I20260925 14:57:38.161795 9538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:38.161844 9538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fb529a890e4dee845380c8cc066c8a, State: Initialized, Role: FOLLOWER
I20260925 14:57:38.161969 9538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fb529a890e4dee845380c8cc066c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38433 } }
I20260925 14:57:38.162065 9538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:38.162060 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:38.162113 9538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:38.162158 9538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:38.162686 9538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fb529a890e4dee845380c8cc066c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38433 } }
I20260925 14:57:38.162804 9538 leader_election.cc:302] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01fb529a890e4dee845380c8cc066c8a; no voters:
I20260925 14:57:38.162963 9538 leader_election.cc:288] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:38.162998 9545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:38.163239 9545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [term 1 LEADER]: Becoming Leader. State: Replica: 01fb529a890e4dee845380c8cc066c8a, State: Running, Role: LEADER
I20260925 14:57:38.163322 9545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fb529a890e4dee845380c8cc066c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38433 } }
I20260925 14:57:38.163357 9538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:38.163661 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01fb529a890e4dee845380c8cc066c8a. Latest consensus state: current_term: 1 leader_uuid: "01fb529a890e4dee845380c8cc066c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fb529a890e4dee845380c8cc066c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38433 } } }
I20260925 14:57:38.163812 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.163663 9548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01fb529a890e4dee845380c8cc066c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fb529a890e4dee845380c8cc066c8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38433 } } }
I20260925 14:57:38.163969 9548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.164419 9555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:38.164932 9555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:38.166507 9555 catalog_manager.cc:1347] Generated new cluster ID: 0ae2af2016bb4b9fbf985268957e126b
I20260925 14:57:38.166560 9555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:38.168345 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.164854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42585 (local address 127.0.0.1:38433)
0925 14:57:38.164887 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.164889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.165089 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:57:38.165184 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.165185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.165194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.165222 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.165438 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.166131 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.167124 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.167408 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.167485 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.168114 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.168115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.168115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:38.168120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.168135 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:38.168140 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.168201 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.168252 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.168273 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 14:57:38.176152 9555 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:38.176633 9555 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:38.180418 9555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01fb529a890e4dee845380c8cc066c8a: Generated new TSK 0
I20260925 14:57:38.180608 9555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:38.184859 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9470
2026-09-25T14:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:38 dist-test-slave-8qt6 krb5kdc[9569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:38 dist-test-slave-8qt6 krb5kdc[9569](info): set up 2 sockets
Sep 25 14:57:38 dist-test-slave-8qt6 krb5kdc[9569](info): commencing operation
krb5kdc: starting...
W20260925 14:57:39.238617 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:39 dist-test-slave-8qt6 krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:39.357959 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:39.438238 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:39.438484 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:39.438520 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:39.439667 9582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:39.439738 9582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:39.439762 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:39.439790 9582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:39.439822 9582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:39.441262 9582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:39.441560 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:39.441790 9582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:39.446794 9582 server_base.cc:1034] running on GCE node
W20260925 14:57:39.446731 9590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:39.446746 9587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:39.446748 9588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:39.447247 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:39.447571 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:39.448732 9582 hybrid_clock.cc:648] HybridClock initialized: now 1790348259448729 us; error 35 us; skew 500 ppm
Sep 25 14:57:39 dist-test-slave-8qt6 krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:39.451893 9582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:39.452389 9582 webserver.cc:460] Webserver started at http://127.0.0.1:38229/ using document root <none> and password file <none>
I20260925 14:57:39.452606 9582 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:39.452674 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:39.452824 9582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:39.453903 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "4cc0cac18bdf4d3998b2938437fa0f46"
format_stamp: "Formatted at 2026-09-25 14:57:39 on dist-test-slave-8qt6"
server_key: "66e03caf69b71f76228c8fe6f11c5164"
server_key_iv: "23d98a98eaf67f1ee9a8757a8e06c7db"
server_key_version: "encryptionkey@0"
I20260925 14:57:39.454270 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "4cc0cac18bdf4d3998b2938437fa0f46"
format_stamp: "Formatted at 2026-09-25 14:57:39 on dist-test-slave-8qt6"
server_key: "66e03caf69b71f76228c8fe6f11c5164"
server_key_iv: "23d98a98eaf67f1ee9a8757a8e06c7db"
server_key_version: "encryptionkey@0"
I20260925 14:57:39.455546 9582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:39.456245 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:39.456413 9582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:39.456514 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "4cc0cac18bdf4d3998b2938437fa0f46"
format_stamp: "Formatted at 2026-09-25 14:57:39 on dist-test-slave-8qt6"
server_key: "66e03caf69b71f76228c8fe6f11c5164"
server_key_iv: "23d98a98eaf67f1ee9a8757a8e06c7db"
server_key_version: "encryptionkey@0"
I20260925 14:57:39.456591 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:39.460858 9582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:39.464380 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:39.464529 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:39.469259 9582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45909
I20260925 14:57:39.469298 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45909 every 8 connection(s)
I20260925 14:57:39.469749 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:39.470907 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:39.472549 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9582
I20260925 14:57:39.472672 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:39.472879 21276 external_mini_cluster.cc:1428] Setting key 4cca1685439d355c08a6a5ccdb367b4e
I20260925 14:57:39.474067 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46: Bootstrap starting.
I20260925 14:57:39.474766 9650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:39.475348 9650 log.cc:824] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:39.475682 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:39.473460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42749 (local address 127.0.0.1:45909)
0925 14:57:39.473648 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:39.473656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:39.473686 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:57:39.474203 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:39.474210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:39.474237 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:39.474354 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:39.475199 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:39.475202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:39.475203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:39.475206 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:39.475240 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:39.475250 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:39.475311 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:39.475390 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:57:39.475444 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260925 14:57:39.476243 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46: No bootstrap required, opened a new log
I20260925 14:57:39.476632 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:39.479190 9650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260925 14:57:39.479377 9650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:39.479444 9650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc0cac18bdf4d3998b2938437fa0f46, State: Initialized, Role: FOLLOWER
I20260925 14:57:39.479583 9650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260925 14:57:39.479713 9650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:39.479733 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:39.478400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38767 (local address 127.0.0.1:45909)
0925 14:57:39.478604 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:39.478606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:39.478688 (+ 82us) server_negotiation.cc:407] Connection header received
0925 14:57:39.478757 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:39.478758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:39.478763 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:39.478828 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:39.479470 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:39.479471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:39.479472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:39.479477 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:39.479493 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:39.479499 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:39.479549 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:39.479649 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:57:39.479665 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260925 14:57:39.479764 9650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:39.479816 9650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:39.480558 9650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260925 14:57:39.480685 9650 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc0cac18bdf4d3998b2938437fa0f46; no voters:
I20260925 14:57:39.480856 9650 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:39.480891 9660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:39.481089 9660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc0cac18bdf4d3998b2938437fa0f46, State: Running, Role: LEADER
I20260925 14:57:39.481143 9650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:39.481244 9660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260925 14:57:39.481668 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc0cac18bdf4d3998b2938437fa0f46. Latest consensus state: current_term: 1 leader_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260925 14:57:39.481884 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:39.481812 9662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc0cac18bdf4d3998b2938437fa0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260925 14:57:39.482132 9662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:39.482494 9669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:39.482990 9669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:39.484735 9669 catalog_manager.cc:1347] Generated new cluster ID: e49d29f36cbc4737aec5de192e9520d3
I20260925 14:57:39.484792 9669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:39.505491 9669 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:39.505733 9669 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:39.513913 9669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cc0cac18bdf4d3998b2938437fa0f46: Generated new TSK 0
I20260925 14:57:39.514242 9669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:39.518271 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9582
2026-09-25T14:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:39 dist-test-slave-8qt6 krb5kdc[9681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:39 dist-test-slave-8qt6 krb5kdc[9681](info): set up 2 sockets
Sep 25 14:57:39 dist-test-slave-8qt6 krb5kdc[9681](info): commencing operation
krb5kdc: starting...
W20260925 14:57:40.549495 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:40 dist-test-slave-8qt6 krb5kdc[9681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:40.667888 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:40.750386 9694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:40.750634 9694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:40.750674 9694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:40.751854 9694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:40.751928 9694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:40.751957 9694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:40.751994 9694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:40.752025 9694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:40.753515 9694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:40.753800 9694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:40.754074 9694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:40.759119 9699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:40.759093 9700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:40.759102 9702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:40.759156 9694 server_base.cc:1034] running on GCE node
I20260925 14:57:40.759595 9694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:40.759925 9694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:40.761099 9694 hybrid_clock.cc:648] HybridClock initialized: now 1790348260761074 us; error 38 us; skew 500 ppm
Sep 25 14:57:40 dist-test-slave-8qt6 krb5kdc[9681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:40.768234 9694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:40.768739 9694 webserver.cc:460] Webserver started at http://127.0.0.1:34987/ using document root <none> and password file <none>
I20260925 14:57:40.768987 9694 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:40.769058 9694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:40.769209 9694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:40.770388 9694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "12ad0aff1203461e8b8f6e8b77b01c3f"
format_stamp: "Formatted at 2026-09-25 14:57:40 on dist-test-slave-8qt6"
server_key: "93722d213fbd34729b67fefd168832e6"
server_key_iv: "36938700484dc57ac61a34bdd4841bc7"
server_key_version: "encryptionkey@0"
I20260925 14:57:40.770754 9694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "12ad0aff1203461e8b8f6e8b77b01c3f"
format_stamp: "Formatted at 2026-09-25 14:57:40 on dist-test-slave-8qt6"
server_key: "93722d213fbd34729b67fefd168832e6"
server_key_iv: "36938700484dc57ac61a34bdd4841bc7"
server_key_version: "encryptionkey@0"
I20260925 14:57:40.772184 9694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:40.773149 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:40.773392 9694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:40.773550 9694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "12ad0aff1203461e8b8f6e8b77b01c3f"
format_stamp: "Formatted at 2026-09-25 14:57:40 on dist-test-slave-8qt6"
server_key: "93722d213fbd34729b67fefd168832e6"
server_key_iv: "36938700484dc57ac61a34bdd4841bc7"
server_key_version: "encryptionkey@0"
I20260925 14:57:40.773634 9694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:40.791431 9694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:40.793493 9694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:40.793660 9694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:40.797709 9694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37667
I20260925 14:57:40.797757 9761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37667 every 8 connection(s)
I20260925 14:57:40.798143 9694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:40.799039 9762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:40.801280 9762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f: Bootstrap starting.
I20260925 14:57:40.801880 9762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:40.802184 9762 log.cc:824] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:40.802445 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9694
I20260925 14:57:40.802590 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:40.802809 21276 external_mini_cluster.cc:1428] Setting key b958070b15971e58b14dd4d73ca218cc
I20260925 14:57:40.803004 9762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f: No bootstrap required, opened a new log
I20260925 14:57:40.804919 9762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 14:57:40.805107 9762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:40.805158 9762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ad0aff1203461e8b8f6e8b77b01c3f, State: Initialized, Role: FOLLOWER
I20260925 14:57:40.805298 9762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 14:57:40.805382 9762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:40.805437 9762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:40.805490 9762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:40.806044 9762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 14:57:40.806164 9762 leader_election.cc:302] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ad0aff1203461e8b8f6e8b77b01c3f; no voters:
I20260925 14:57:40.806321 9762 leader_election.cc:288] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:40.806350 9767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:40.806465 9767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [term 1 LEADER]: Becoming Leader. State: Replica: 12ad0aff1203461e8b8f6e8b77b01c3f, State: Running, Role: LEADER
I20260925 14:57:40.806587 9762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:40.806617 9767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 14:57:40.807022 9767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ad0aff1203461e8b8f6e8b77b01c3f. Latest consensus state: current_term: 1 leader_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } } }
I20260925 14:57:40.807219 9767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:40.807163 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ad0aff1203461e8b8f6e8b77b01c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } } }
I20260925 14:57:40.807426 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:40.807837 9776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:40.808334 9776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:40.810055 9776 catalog_manager.cc:1347] Generated new cluster ID: 79f3a8051d57476289161659ba4f7b43
I20260925 14:57:40.810109 9776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:40 dist-test-slave-8qt6 krb5kdc[9681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:40.814926 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:40.803372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:34029 (local address 127.0.0.1:37667)
0925 14:57:40.803572 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:40.803583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:40.803617 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:57:40.804254 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:40.804262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:40.804313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:40.804464 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:40.805089 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:40.806543 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:40.807640 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:40.808251 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:40.808663 (+ 412us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:40.811143 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:40.811147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:40.811148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:40.811159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:40.813048 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:40.813500 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:40.813503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:40.813504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:40.813596 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:40.813808 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:40.813809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:40.813809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:40.814085 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:40.814571 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:40.814649 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:57:40.814738 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260925 14:57:40.815757 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:40.824124 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:40.817124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:36179 (local address 127.0.0.1:37667)
0925 14:57:40.817158 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:40.817159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:40.817260 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:57:40.817335 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:40.817336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:40.817343 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:40.817367 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:40.818013 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:40.818726 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:40.819666 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:40.819986 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:40.820029 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:40.821473 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:40.821475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:40.821476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:40.821483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:40.822930 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:40.823323 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:40.823324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:40.823324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:40.823369 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:40.823566 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:40.823567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:40.823567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:40.823674 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:40.823742 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:40.823960 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:57:40.824034 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:57:40.825213 9776 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:40.825799 9776 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:40.830651 9776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ad0aff1203461e8b8f6e8b77b01c3f: Generated new TSK 0
I20260925 14:57:40.830837 9776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:40.835372 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9694
W20260925 14:57:40.839740 9679 connection.cc:535] client connection to 127.0.0.1:37667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37667): Connection reset by peer (error 104)
2026-09-25T14:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:40 dist-test-slave-8qt6 krb5kdc[9793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:40 dist-test-slave-8qt6 krb5kdc[9793](info): set up 2 sockets
Sep 25 14:57:40 dist-test-slave-8qt6 krb5kdc[9793](info): commencing operation
krb5kdc: starting...
W20260925 14:57:41.885792 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:41 dist-test-slave-8qt6 krb5kdc[9793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:42.008663 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:42.089525 9806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:42.089807 9806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:42.089849 9806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:42.091054 9806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:42.091130 9806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:42.091153 9806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:42.091174 9806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:42.091204 9806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:42.092701 9806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:42.092981 9806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:42.093214 9806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:42.098181 9812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.098313 9806 server_base.cc:1034] running on GCE node
W20260925 14:57:42.098137 9814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.098137 9811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.098726 9806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:42.099097 9806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:42.100274 9806 hybrid_clock.cc:648] HybridClock initialized: now 1790348262100258 us; error 28 us; skew 500 ppm
Sep 25 14:57:42 dist-test-slave-8qt6 krb5kdc[9793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:42.102855 9806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:42.103323 9806 webserver.cc:460] Webserver started at http://127.0.0.1:45139/ using document root <none> and password file <none>
I20260925 14:57:42.103518 9806 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:42.103559 9806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:42.103647 9806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:42.104661 9806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "59d57073adb44ea9baad6969d79c46a9"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-8qt6"
server_key: "f427c54166e4f3af59e37802ebe48901"
server_key_iv: "16ad12d4cb6002a436f28bc4b66b7699"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.104974 9806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "59d57073adb44ea9baad6969d79c46a9"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-8qt6"
server_key: "f427c54166e4f3af59e37802ebe48901"
server_key_iv: "16ad12d4cb6002a436f28bc4b66b7699"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.106348 9806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:57:42.107084 9821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.107275 9806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:42.107360 9806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "59d57073adb44ea9baad6969d79c46a9"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-8qt6"
server_key: "f427c54166e4f3af59e37802ebe48901"
server_key_iv: "16ad12d4cb6002a436f28bc4b66b7699"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.107411 9806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:42.134074 9806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:42.136415 9806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:42.136528 9806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:42.140555 9806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38699
I20260925 14:57:42.140615 9873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38699 every 8 connection(s)
I20260925 14:57:42.140964 9806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:42.141876 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:42.143422 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9806
I20260925 14:57:42.143560 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:42.143755 21276 external_mini_cluster.cc:1428] Setting key de0def6b4cced98573c95228c1cea32b
I20260925 14:57:42.144502 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9: Bootstrap starting.
I20260925 14:57:42.145198 9874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:42.145573 9874 log.cc:824] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:42.146322 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9: No bootstrap required, opened a new log
I20260925 14:57:42.148505 9874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d57073adb44ea9baad6969d79c46a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260925 14:57:42.148710 9874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:42.148761 9874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d57073adb44ea9baad6969d79c46a9, State: Initialized, Role: FOLLOWER
I20260925 14:57:42.148892 9874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d57073adb44ea9baad6969d79c46a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260925 14:57:42.148972 9874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:42.149031 9874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:42.149084 9874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:42.149741 9874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d57073adb44ea9baad6969d79c46a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260925 14:57:42.149863 9874 leader_election.cc:302] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d57073adb44ea9baad6969d79c46a9; no voters:
I20260925 14:57:42.149993 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.144324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:38889 (local address 127.0.0.1:38699)
0925 14:57:42.144519 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.144529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.144563 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:57:42.145171 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.145181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.145248 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.145442 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.145627 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.146916 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.147680 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.148371 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.148484 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.149204 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.149211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.149211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:42.149445 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.149504 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:42.149516 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.149610 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.149716 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.149743 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260925 14:57:42.150063 9874 leader_election.cc:288] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:42.150091 9879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:42.150200 9879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [term 1 LEADER]: Becoming Leader. State: Replica: 59d57073adb44ea9baad6969d79c46a9, State: Running, Role: LEADER
I20260925 14:57:42.150308 9874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:42.150488 9879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d57073adb44ea9baad6969d79c46a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260925 14:57:42.150979 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:42.151008 9879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59d57073adb44ea9baad6969d79c46a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d57073adb44ea9baad6969d79c46a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } } }
I20260925 14:57:42.151067 9880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d57073adb44ea9baad6969d79c46a9. Latest consensus state: current_term: 1 leader_uuid: "59d57073adb44ea9baad6969d79c46a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d57073adb44ea9baad6969d79c46a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } } }
I20260925 14:57:42.151094 9879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:42.151116 9880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:42.152261 9895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:42.152307 9895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:42.152351 9888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:42.152503 9888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:42.154091 9888 catalog_manager.cc:1347] Generated new cluster ID: a6d578765e0146499271fbb045d1e8c8
I20260925 14:57:42.154142 9888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:42.156307 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.152610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:49515 (local address 127.0.0.1:38699)
0925 14:57:42.152650 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.152652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.153234 (+ 582us) server_negotiation.cc:407] Connection header received
0925 14:57:42.153342 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.153343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.153355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.153441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.153620 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.154334 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.155073 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.155302 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.155377 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.156066 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.156067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.156067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:42.156073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.156087 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:42.156091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.156144 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.156204 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.156214 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:42.159089 9888 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:42.159590 9888 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:42.163327 9888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59d57073adb44ea9baad6969d79c46a9: Generated new TSK 0
I20260925 14:57:42.163518 9888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:42.168576 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9806
W20260925 14:57:42.175197 9791 connection.cc:535] client connection to 127.0.0.1:38699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38699): Connection reset by peer (error 104)
2026-09-25T14:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:42 dist-test-slave-8qt6 krb5kdc[9905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:42 dist-test-slave-8qt6 krb5kdc[9905](info): set up 2 sockets
Sep 25 14:57:42 dist-test-slave-8qt6 krb5kdc[9905](info): commencing operation
krb5kdc: starting...
W20260925 14:57:43.205572 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:43 dist-test-slave-8qt6 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:43.324653 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:43.411022 9918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:43.411218 9918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:43.411240 9918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:43.412321 9918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:43.412361 9918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:43.412376 9918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:43.412389 9918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:43.412401 9918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:43.413806 9918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:43.414040 9918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:43.414219 9918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:43.418936 9923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.419026 9924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.419097 9918 server_base.cc:1034] running on GCE node
W20260925 14:57:43.418886 9926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.419349 9918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:43.419580 9918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:43.420720 9918 hybrid_clock.cc:648] HybridClock initialized: now 1790348263420707 us; error 29 us; skew 500 ppm
Sep 25 14:57:43 dist-test-slave-8qt6 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:43.423280 9918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:43.423724 9918 webserver.cc:460] Webserver started at http://127.0.0.1:42815/ using document root <none> and password file <none>
I20260925 14:57:43.423997 9918 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:43.424080 9918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:43.424223 9918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:43.425236 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "244fb13bcd234d3ea6da26fe1498f34e"
format_stamp: "Formatted at 2026-09-25 14:57:43 on dist-test-slave-8qt6"
server_key: "8d0d76e7b657113efa70c07236be52ab"
server_key_iv: "b41708beed12ec08cf78b67bf1b2d016"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.425618 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "244fb13bcd234d3ea6da26fe1498f34e"
format_stamp: "Formatted at 2026-09-25 14:57:43 on dist-test-slave-8qt6"
server_key: "8d0d76e7b657113efa70c07236be52ab"
server_key_iv: "b41708beed12ec08cf78b67bf1b2d016"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.427022 9918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:43.427734 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.427892 9918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:57:43.428000 9918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "244fb13bcd234d3ea6da26fe1498f34e"
format_stamp: "Formatted at 2026-09-25 14:57:43 on dist-test-slave-8qt6"
server_key: "8d0d76e7b657113efa70c07236be52ab"
server_key_iv: "b41708beed12ec08cf78b67bf1b2d016"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.428076 9918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:43.431351 9918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:43.433575 9918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:43.433725 9918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:43.437467 9918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41131
I20260925 14:57:43.437497 9985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41131 every 8 connection(s)
I20260925 14:57:43.437835 9918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:43.438661 9986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:43.439281 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 9918
I20260925 14:57:43.439410 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:43.439599 21276 external_mini_cluster.cc:1428] Setting key a7275ccd9c7d3b14d05aea581c947881
I20260925 14:57:43.442018 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e: Bootstrap starting.
I20260925 14:57:43.442256 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.440100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:59455 (local address 127.0.0.1:41131)
0925 14:57:43.440526 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.440535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.440566 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:57:43.441057 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.441065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.441101 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.441253 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.441850 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.441853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.441853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:43.441857 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.441892 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:43.441902 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.441959 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.442053 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.442102 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":279,"threads_started":1}
I20260925 14:57:43.442751 9986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:43.443112 9986 log.cc:824] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:43.443380 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:43.443801 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e: No bootstrap required, opened a new log
I20260925 14:57:43.445894 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.444784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:38993 (local address 127.0.0.1:41131)
0925 14:57:43.444821 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.444822 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.444946 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:57:43.445020 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.445021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.445026 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.445097 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.445683 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.445685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.445685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:43.445689 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.445701 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:43.445705 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.445761 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.445817 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.445831 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:43.446043 9986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244fb13bcd234d3ea6da26fe1498f34e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260925 14:57:43.446198 9986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:43.446223 9986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244fb13bcd234d3ea6da26fe1498f34e, State: Initialized, Role: FOLLOWER
I20260925 14:57:43.446360 9986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244fb13bcd234d3ea6da26fe1498f34e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260925 14:57:43.446484 9986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:43.446530 9986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:43.446580 9986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:43.447124 9986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244fb13bcd234d3ea6da26fe1498f34e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260925 14:57:43.447239 9986 leader_election.cc:302] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244fb13bcd234d3ea6da26fe1498f34e; no voters:
I20260925 14:57:43.447432 9986 leader_election.cc:288] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:43.447466 9996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:43.447633 9996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [term 1 LEADER]: Becoming Leader. State: Replica: 244fb13bcd234d3ea6da26fe1498f34e, State: Running, Role: LEADER
I20260925 14:57:43.447680 9986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:43.447777 9996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244fb13bcd234d3ea6da26fe1498f34e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260925 14:57:43.448156 9997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "244fb13bcd234d3ea6da26fe1498f34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244fb13bcd234d3ea6da26fe1498f34e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } } }
I20260925 14:57:43.448159 9998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244fb13bcd234d3ea6da26fe1498f34e. Latest consensus state: current_term: 1 leader_uuid: "244fb13bcd234d3ea6da26fe1498f34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244fb13bcd234d3ea6da26fe1498f34e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } } }
I20260925 14:57:43.448324 9997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:43.448333 9998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:43.448573 10005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:43.449100 10005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:43.450774 10005 catalog_manager.cc:1347] Generated new cluster ID: 75832f45cad541d88545a768205e7e29
I20260925 14:57:43.450826 10005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:43.468151 10005 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:43.468321 10005 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:43.482456 10005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244fb13bcd234d3ea6da26fe1498f34e: Generated new TSK 0
I20260925 14:57:43.482697 10005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:43.490320 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 9918
2026-09-25T14:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:43 dist-test-slave-8qt6 krb5kdc[10017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:43 dist-test-slave-8qt6 krb5kdc[10017](info): set up 2 sockets
Sep 25 14:57:43 dist-test-slave-8qt6 krb5kdc[10017](info): commencing operation
krb5kdc: starting...
W20260925 14:57:44.529628 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:44 dist-test-slave-8qt6 krb5kdc[10017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:44.649013 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:44.729023 10030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.729276 10030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.729316 10030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.730505 10030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:44.730579 10030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.730604 10030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.730633 10030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:44.730664 10030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:44.732115 10030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:44.732399 10030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.732622 10030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:44.737501 10030 server_base.cc:1034] running on GCE node
W20260925 14:57:44.737478 10035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.737653 10036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.737458 10038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.737911 10030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.738211 10030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.739379 10030 hybrid_clock.cc:648] HybridClock initialized: now 1790348264739381 us; error 44 us; skew 500 ppm
Sep 25 14:57:44 dist-test-slave-8qt6 krb5kdc[10017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.741840 10030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:44.742261 10030 webserver.cc:460] Webserver started at http://127.0.0.1:33693/ using document root <none> and password file <none>
I20260925 14:57:44.742480 10030 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.742555 10030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.742707 10030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:44.743784 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "005b4fa24e894a5180384bef3c9eedc3"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-8qt6"
server_key: "4c19a36a1bac08ecee162a04198aba94"
server_key_iv: "dcca453ee2b08f1ab7e931600d6a3427"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.744158 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "005b4fa24e894a5180384bef3c9eedc3"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-8qt6"
server_key: "4c19a36a1bac08ecee162a04198aba94"
server_key_iv: "dcca453ee2b08f1ab7e931600d6a3427"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.745525 10030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:44.746295 10045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.746476 10030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:44.746585 10030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "005b4fa24e894a5180384bef3c9eedc3"
format_stamp: "Formatted at 2026-09-25 14:57:44 on dist-test-slave-8qt6"
server_key: "4c19a36a1bac08ecee162a04198aba94"
server_key_iv: "dcca453ee2b08f1ab7e931600d6a3427"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.746675 10030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.758520 10030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.760823 10030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.760996 10030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.765076 10030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34231
I20260925 14:57:44.765129 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34231 every 8 connection(s)
I20260925 14:57:44.765587 10030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:44.766496 10098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:44.768764 10098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3: Bootstrap starting.
I20260925 14:57:44.769382 10098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:44.769745 10098 log.cc:824] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:44.770533 10098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3: No bootstrap required, opened a new log
I20260925 14:57:44.772272 10098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005b4fa24e894a5180384bef3c9eedc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260925 14:57:44.772431 10098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:44.772456 10098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005b4fa24e894a5180384bef3c9eedc3, State: Initialized, Role: FOLLOWER
I20260925 14:57:44.772627 10098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005b4fa24e894a5180384bef3c9eedc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260925 14:57:44.772691 10098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:44.772712 10098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:44.772773 10098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:44.773331 10098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005b4fa24e894a5180384bef3c9eedc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260925 14:57:44.773474 10098 leader_election.cc:302] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 005b4fa24e894a5180384bef3c9eedc3; no voters:
I20260925 14:57:44.773686 10098 leader_election.cc:288] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:44.773753 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10030
I20260925 14:57:44.773726 10101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:44.773874 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:44.773924 10101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [term 1 LEADER]: Becoming Leader. State: Replica: 005b4fa24e894a5180384bef3c9eedc3, State: Running, Role: LEADER
I20260925 14:57:44.773970 10098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:44.774009 10101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005b4fa24e894a5180384bef3c9eedc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260925 14:57:44.774056 21276 external_mini_cluster.cc:1428] Setting key 66338940318622c6c43c002e33a090be
I20260925 14:57:44.774464 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 005b4fa24e894a5180384bef3c9eedc3. Latest consensus state: current_term: 1 leader_uuid: "005b4fa24e894a5180384bef3c9eedc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005b4fa24e894a5180384bef3c9eedc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } } }
I20260925 14:57:44.774554 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:44.774715 10102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "005b4fa24e894a5180384bef3c9eedc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005b4fa24e894a5180384bef3c9eedc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } } }
I20260925 14:57:44.774766 10102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:44.775009 10111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:44.775512 10111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:44.777810 10111 catalog_manager.cc:1347] Generated new cluster ID: ecf85adf012b42f2915a118f88a16f84
I20260925 14:57:44.777868 10111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:44 dist-test-slave-8qt6 krb5kdc[10017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:57:44.786167 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.774766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:56359 (local address 127.0.0.1:34231)
0925 14:57:44.774950 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.774962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.775159 (+ 197us) server_negotiation.cc:407] Connection header received
0925 14:57:44.775357 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.775364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.775424 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.776305 (+ 881us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.776402 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.777664 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.778702 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.779335 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.779453 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.782402 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.782406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.782407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.782419 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.784395 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.784856 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.784859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.784860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.784978 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.785230 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.785231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.785231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.785556 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.785716 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.785839 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.785935 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260925 14:57:44.787385 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:44.788398 10111 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:44.789130 10111 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:44.794706 10111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 005b4fa24e894a5180384bef3c9eedc3: Generated new TSK 0
I20260925 14:57:44.794888 10111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:44.796314 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.788904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:52291 (local address 127.0.0.1:34231)
0925 14:57:44.788938 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.788939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.789180 (+ 241us) server_negotiation.cc:407] Connection header received
0925 14:57:44.789275 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.789276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.789286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.789314 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:44.790017 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.790786 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.791770 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.792082 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.792116 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.793562 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:44.793563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:44.793563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:44.793570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:44.795132 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.795515 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.795516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.795517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.795569 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:44.795810 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:44.795812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:44.795812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:44.795917 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:44.796027 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.796158 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.796220 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:57:44.798328 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10030
W20260925 14:57:44.802294 10015 connection.cc:535] client connection to 127.0.0.1:34231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34231): Connection reset by peer (error 104)
2026-09-25T14:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:44 dist-test-slave-8qt6 krb5kdc[10129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:44 dist-test-slave-8qt6 krb5kdc[10129](info): set up 2 sockets
Sep 25 14:57:44 dist-test-slave-8qt6 krb5kdc[10129](info): commencing operation
krb5kdc: starting...
W20260925 14:57:45.821573 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:45 dist-test-slave-8qt6 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:45.942518 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:46.027375 10142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:46.027591 10142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:46.027616 10142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:46.028913 10142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:46.028970 10142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:46.028985 10142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:46.029026 10142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:46.029055 10142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:46.030620 10142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:46.030879 10142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:46.031114 10142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:46.036013 10142 server_base.cc:1034] running on GCE node
W20260925 14:57:46.035962 10148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:46.035955 10147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:46.035962 10150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:46.036439 10142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:46.036782 10142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:46.037940 10142 hybrid_clock.cc:648] HybridClock initialized: now 1790348266037937 us; error 34 us; skew 500 ppm
Sep 25 14:57:46 dist-test-slave-8qt6 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:46.046064 10142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:46.046571 10142 webserver.cc:460] Webserver started at http://127.0.0.1:38975/ using document root <none> and password file <none>
I20260925 14:57:46.046808 10142 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:46.046880 10142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:46.047049 10142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:46.048183 10142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "6e5165333dc541e68025fadb156bb364"
format_stamp: "Formatted at 2026-09-25 14:57:46 on dist-test-slave-8qt6"
server_key: "23670422562cb2daac4412af5d413998"
server_key_iv: "50d51692478cdf391f4e4c9903ea4fed"
server_key_version: "encryptionkey@0"
I20260925 14:57:46.048571 10142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "6e5165333dc541e68025fadb156bb364"
format_stamp: "Formatted at 2026-09-25 14:57:46 on dist-test-slave-8qt6"
server_key: "23670422562cb2daac4412af5d413998"
server_key_iv: "50d51692478cdf391f4e4c9903ea4fed"
server_key_version: "encryptionkey@0"
I20260925 14:57:46.050020 10142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:46.050839 10157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:46.051075 10142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:46.051186 10142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "6e5165333dc541e68025fadb156bb364"
format_stamp: "Formatted at 2026-09-25 14:57:46 on dist-test-slave-8qt6"
server_key: "23670422562cb2daac4412af5d413998"
server_key_iv: "50d51692478cdf391f4e4c9903ea4fed"
server_key_version: "encryptionkey@0"
I20260925 14:57:46.051261 10142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:46.072886 10142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:46.074803 10142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:46.074985 10142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:46.079298 10142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37013
I20260925 14:57:46.079339 10209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37013 every 8 connection(s)
I20260925 14:57:46.079710 10142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:46.080572 10210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:46.082772 10210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364: Bootstrap starting.
I20260925 14:57:46.083498 10210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:46.083760 10210 log.cc:824] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:46.084461 10210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364: No bootstrap required, opened a new log
I20260925 14:57:46.086175 10210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5165333dc541e68025fadb156bb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37013 } }
I20260925 14:57:46.086313 10210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:46.086335 10210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5165333dc541e68025fadb156bb364, State: Initialized, Role: FOLLOWER
I20260925 14:57:46.086439 10210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5165333dc541e68025fadb156bb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37013 } }
I20260925 14:57:46.086493 10210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:46.086514 10210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:46.086581 10210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:46.087116 10210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5165333dc541e68025fadb156bb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37013 } }
I20260925 14:57:46.087241 10210 leader_election.cc:302] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e5165333dc541e68025fadb156bb364; no voters:
I20260925 14:57:46.087430 10210 leader_election.cc:288] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:46.087464 10213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:46.087659 10213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5165333dc541e68025fadb156bb364, State: Running, Role: LEADER
I20260925 14:57:46.087700 10210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:46.087782 10213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5165333dc541e68025fadb156bb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37013 } }
I20260925 14:57:46.088367 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10142
I20260925 14:57:46.088392 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e5165333dc541e68025fadb156bb364. Latest consensus state: current_term: 1 leader_uuid: "6e5165333dc541e68025fadb156bb364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5165333dc541e68025fadb156bb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37013 } } }
I20260925 14:57:46.088476 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:46.088500 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:46.088706 21276 external_mini_cluster.cc:1428] Setting key 094d2e087c0698f0866e3885776b13b2
I20260925 14:57:46.088713 10218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:46.088694 10214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e5165333dc541e68025fadb156bb364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5165333dc541e68025fadb156bb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37013 } } }
I20260925 14:57:46.088838 10214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:46.089545 10218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:46.094755 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:46.089357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:40875 (local address 127.0.0.1:37013)
0925 14:57:46.089556 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:46.089577 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:46.089610 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:57:46.089727 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:46.089734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:46.089793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:46.090224 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:46.090352 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:46.091759 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:46.092533 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:46.093165 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:46.093282 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:46.094063 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:46.094067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:46.094067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:46.094281 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:46.094320 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:46.094336 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:46.094425 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:46.094500 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:57:46.094531 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260925 14:57:46.095018 10218 catalog_manager.cc:1347] Generated new cluster ID: fe4e2bce62964d98950b092693b5eab5
I20260925 14:57:46.095094 10218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:46.095716 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:46.100807 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:46.097461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:52299 (local address 127.0.0.1:37013)
0925 14:57:46.097498 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:46.097500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:46.097509 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:46.097528 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:46.097529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:46.097540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:46.097600 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:46.097745 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:46.098508 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:46.099482 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:46.099821 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:46.099897 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:46.100557 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:46.100559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:46.100560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:46.100564 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:46.100579 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:46.100584 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:46.100642 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:46.100720 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:57:46.100734 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:57:46.123625 10218 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:46.124176 10218 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:46.128567 10218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e5165333dc541e68025fadb156bb364: Generated new TSK 0
I20260925 14:57:46.128752 10218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:46.131222 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10142
W20260925 14:57:46.135463 10127 connection.cc:535] client connection to 127.0.0.1:37013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37013): Connection reset by peer (error 104)
2026-09-25T14:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:46 dist-test-slave-8qt6 krb5kdc[10241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:46 dist-test-slave-8qt6 krb5kdc[10241](info): set up 2 sockets
Sep 25 14:57:46 dist-test-slave-8qt6 krb5kdc[10241](info): commencing operation
krb5kdc: starting...
W20260925 14:57:47.165657 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:47 dist-test-slave-8qt6 krb5kdc[10241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:57:47.282297 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:47.364513 10254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:47.364717 10254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:47.364742 10254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:47.365993 10254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:47.366042 10254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:47.366057 10254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:47.366071 10254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:47.366083 10254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:47.367538 10254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:47.367780 10254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:47.367997 10254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:47.372772 10254 server_base.cc:1034] running on GCE node
W20260925 14:57:47.372675 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:47.372677 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:47.372838 10259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:47.373157 10254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:47.373514 10254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:47.374660 10254 hybrid_clock.cc:648] HybridClock initialized: now 1790348267374637 us; error 47 us; skew 500 ppm
Sep 25 14:57:47 dist-test-slave-8qt6 krb5kdc[10241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:47.377152 10254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:57:47.377653 10254 webserver.cc:460] Webserver started at http://127.0.0.1:46215/ using document root <none> and password file <none>
I20260925 14:57:47.377874 10254 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:47.377940 10254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:47.378095 10254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:47.379206 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0"
format_stamp: "Formatted at 2026-09-25 14:57:47 on dist-test-slave-8qt6"
server_key: "bd05870593effd75f18ba7612147d623"
server_key_iv: "fca7dbb1f97e3e1af5affbd20d5702de"
server_key_version: "encryptionkey@0"
I20260925 14:57:47.379571 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0"
format_stamp: "Formatted at 2026-09-25 14:57:47 on dist-test-slave-8qt6"
server_key: "bd05870593effd75f18ba7612147d623"
server_key_iv: "fca7dbb1f97e3e1af5affbd20d5702de"
server_key_version: "encryptionkey@0"
I20260925 14:57:47.380890 10254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:47.381769 10269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:47.381924 10254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:47.382022 10254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0"
format_stamp: "Formatted at 2026-09-25 14:57:47 on dist-test-slave-8qt6"
server_key: "bd05870593effd75f18ba7612147d623"
server_key_iv: "fca7dbb1f97e3e1af5affbd20d5702de"
server_key_version: "encryptionkey@0"
I20260925 14:57:47.382107 10254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:47.385269 10254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:47.387542 10254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:47.387696 10254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:47.391537 10254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44617
I20260925 14:57:47.391568 10321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44617 every 8 connection(s)
I20260925 14:57:47.391889 10254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:47.392877 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:47.395339 10322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0: Bootstrap starting.
I20260925 14:57:47.395908 10322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:47.396135 10322 log.cc:824] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:47.396631 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10254
I20260925 14:57:47.396752 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:47.396927 10322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0: No bootstrap required, opened a new log
I20260925 14:57:47.396967 21276 external_mini_cluster.cc:1428] Setting key 972fad2fb9c5d75fdba18d4b0b6dfc09
I20260925 14:57:47.399468 10322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260925 14:57:47.399632 10322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:47.399655 10322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b60bdd79eb34fbeb76a37bff0a8b6c0, State: Initialized, Role: FOLLOWER
I20260925 14:57:47.399791 10322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260925 14:57:47.399869 10322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:47.399914 10322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:47.399943 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:47.397515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:52743 (local address 127.0.0.1:44617)
0925 14:57:47.397672 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:47.397681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:47.397714 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:57:47.398371 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:47.398378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:47.398404 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:47.398528 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:47.399201 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:47.399204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:47.399204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:47.399208 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:47.399247 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:47.399256 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:47.399313 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:47.399401 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 14:57:47.399461 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260925 14:57:47.399963 10322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:47.400769 10322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260925 14:57:47.400889 10322 leader_election.cc:302] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b60bdd79eb34fbeb76a37bff0a8b6c0; no voters:
I20260925 14:57:47.401067 10322 leader_election.cc:288] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:47.401093 10327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:47.401089 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:47.401204 10327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [term 1 LEADER]: Becoming Leader. State: Replica: 2b60bdd79eb34fbeb76a37bff0a8b6c0, State: Running, Role: LEADER
I20260925 14:57:47.401324 10322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:47.401350 10327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260925 14:57:47.401762 10327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b60bdd79eb34fbeb76a37bff0a8b6c0. Latest consensus state: current_term: 1 leader_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } } }
I20260925 14:57:47.402041 10327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:47.401948 10328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b60bdd79eb34fbeb76a37bff0a8b6c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } } }
I20260925 14:57:47.402483 10328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:47.402624 10338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:47.403232 10338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:47.404098 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:47.402596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:41339 (local address 127.0.0.1:44617)
0925 14:57:47.402634 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:47.402637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:47.403073 (+ 436us) server_negotiation.cc:407] Connection header received
0925 14:57:47.403121 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:47.403122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:47.403128 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:47.403182 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:47.403824 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:47.403825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:47.403825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:47.403830 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:47.403846 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:47.403850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:47.403911 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:47.404017 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:57:47.404032 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:47.405807 10338 catalog_manager.cc:1347] Generated new cluster ID: 1597b13c70ee451f810d49e049897f8d
I20260925 14:57:47.405860 10338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:47.422564 10338 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:47.422756 10338 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:47.429441 10338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b60bdd79eb34fbeb76a37bff0a8b6c0: Generated new TSK 0
I20260925 14:57:47.429643 10338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:47.431311 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10254
2026-09-25T14:57:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:47 dist-test-slave-8qt6 krb5kdc[10353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:47 dist-test-slave-8qt6 krb5kdc[10353](info): set up 2 sockets
Sep 25 14:57:47 dist-test-slave-8qt6 krb5kdc[10353](info): commencing operation
krb5kdc: starting...
W20260925 14:57:48.477677 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:48 dist-test-slave-8qt6 krb5kdc[10353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:48.593873 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34633
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:48.675285 10366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:48.675642 10366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:48.675731 10366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:48.676898 10366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:48.676977 10366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:48.677004 10366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:48.677035 10366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:48.677059 10366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:48.678587 10366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:34633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:48.678859 10366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:48.679090 10366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:48.684043 10371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:48.684216 10366 server_base.cc:1034] running on GCE node
W20260925 14:57:48.684016 10372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:48.684016 10374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:48.684588 10366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:48.684871 10366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:48.686033 10366 hybrid_clock.cc:648] HybridClock initialized: now 1790348268686018 us; error 46 us; skew 500 ppm
Sep 25 14:57:48 dist-test-slave-8qt6 krb5kdc[10353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:48.688390 10366 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:48.688859 10366 webserver.cc:460] Webserver started at http://127.20.199.62:42185/ using document root <none> and password file <none>
I20260925 14:57:48.689076 10366 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:48.689142 10366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:48.689289 10366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:48.690380 10366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0"
format_stamp: "Formatted at 2026-09-25 14:57:48 on dist-test-slave-8qt6"
server_key: "c56b9a74def046c376e58a09b009190f"
server_key_iv: "ceff58db76bac6795b5d7c1083e74c17"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.690749 10366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0"
format_stamp: "Formatted at 2026-09-25 14:57:48 on dist-test-slave-8qt6"
server_key: "c56b9a74def046c376e58a09b009190f"
server_key_iv: "ceff58db76bac6795b5d7c1083e74c17"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.692046 10366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:48.692816 10381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:48.693010 10366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:48.693109 10366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0"
format_stamp: "Formatted at 2026-09-25 14:57:48 on dist-test-slave-8qt6"
server_key: "c56b9a74def046c376e58a09b009190f"
server_key_iv: "ceff58db76bac6795b5d7c1083e74c17"
server_key_version: "encryptionkey@0"
I20260925 14:57:48.693195 10366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:48.716396 10366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:48.718525 10366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:48.718698 10366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:48.722688 10366 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:34633
I20260925 14:57:48.722725 10433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:34633 every 8 connection(s)
I20260925 14:57:48.723104 10366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:48.723925 10434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:48.726204 10434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0: Bootstrap starting.
I20260925 14:57:48.726787 10434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:48.727072 10434 log.cc:824] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:48.727752 10434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0: No bootstrap required, opened a new log
I20260925 14:57:48.728363 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10366
I20260925 14:57:48.728493 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:48.728686 21276 external_mini_cluster.cc:1428] Setting key ef41b05ef4da6ce95ccfa0239a233325
I20260925 14:57:48.729907 10434 raft_consensus.cc:348] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34633 } }
I20260925 14:57:48.730052 10434 raft_consensus.cc:374] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:48.730075 10434 raft_consensus.cc:729] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36e7f8c6c3e48ac84ca2d4394a9d2d0, State: Initialized, Role: FOLLOWER
I20260925 14:57:48.730201 10434 consensus_queue.cc:260] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34633 } }
I20260925 14:57:48.730279 10434 raft_consensus.cc:388] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:48.730301 10434 raft_consensus.cc:482] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:48.730366 10434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:48.731204 10434 raft_consensus.cc:504] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34633 } }
I20260925 14:57:48.731320 10434 leader_election.cc:302] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c36e7f8c6c3e48ac84ca2d4394a9d2d0; no voters:
I20260925 14:57:48.731514 10434 leader_election.cc:288] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:48.731554 10439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:48.731658 10439 raft_consensus.cc:686] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [term 1 LEADER]: Becoming Leader. State: Replica: c36e7f8c6c3e48ac84ca2d4394a9d2d0, State: Running, Role: LEADER
I20260925 14:57:48.731776 10434 sys_catalog.cc:564] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:48.731796 10439 consensus_queue.cc:237] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34633 } }
I20260925 14:57:48.732121 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c36e7f8c6c3e48ac84ca2d4394a9d2d0. Latest consensus state: current_term: 1 leader_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34633 } } }
I20260925 14:57:48.732132 10440 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e7f8c6c3e48ac84ca2d4394a9d2d0" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 34633 } } }
I20260925 14:57:48.732241 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:48.732331 10440 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:48.733632 10454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:48.733731 10454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:48.733932 10448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:48.734206 10448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:48.735888 10448 catalog_manager.cc:1347] Generated new cluster ID: 3249c344be0648cbb292f45ddab8e041
I20260925 14:57:48.735937 10448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:48 dist-test-slave-8qt6 krb5kdc[10353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:48.741580 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.729258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44945 (local address 127.20.199.62:34633)
0925 14:57:48.729456 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:48.729466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:48.729503 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:57:48.729946 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:48.729954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:48.730008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:48.730213 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:48.730995 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.732238 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.733114 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.733783 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.733930 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.736776 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:48.736780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:48.736780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:48.736792 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:48.738903 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.739376 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.739379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.739379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.739496 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.739769 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.739770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.739770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.740087 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:48.741086 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:48.741179 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:57:48.741285 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260925 14:57:48.742417 10448 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:48.742587 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:48.743121 10448 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:48.750519 10448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c36e7f8c6c3e48ac84ca2d4394a9d2d0: Generated new TSK 0
I20260925 14:57:48.750732 10448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:48.751611 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.744090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43247 (local address 127.20.199.62:34633)
0925 14:57:48.744125 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:48.744127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:48.744152 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:57:48.744209 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:48.744210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:48.744220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:48.744288 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:48.744861 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.745678 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.746670 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:48.747008 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:48.747082 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:48.748436 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:48.748437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:48.748437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:48.748444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:48.750207 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.750728 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.750730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.750730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.750819 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:48.751092 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:48.751093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:48.751094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:48.751210 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:48.751327 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:48.751449 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 14:57:48.751515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:48.753613 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10366
W20260925 14:57:48.757874 10351 connection.cc:535] client connection to 127.20.199.62:34633 recv error: Network error: failed to read from TLS socket (remote: 127.20.199.62:34633): Connection reset by peer (error 104)
2026-09-25T14:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:48 dist-test-slave-8qt6 krb5kdc[10465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:48 dist-test-slave-8qt6 krb5kdc[10465](info): set up 2 sockets
Sep 25 14:57:48 dist-test-slave-8qt6 krb5kdc[10465](info): commencing operation
krb5kdc: starting...
W20260925 14:57:49.797544 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:49 dist-test-slave-8qt6 krb5kdc[10465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:49.916910 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:41475
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:50.002630 10478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:50.002892 10478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:50.002940 10478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:50.004136 10478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:50.004208 10478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:50.004233 10478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:50.004262 10478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:50.004292 10478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:50.005861 10478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:41475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:50.006152 10478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:50.006389 10478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:50.011423 10483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:50.011605 10478 server_base.cc:1034] running on GCE node
W20260925 14:57:50.011435 10484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:50.011499 10486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:50.011952 10478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:50.012279 10478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:50.013485 10478 hybrid_clock.cc:648] HybridClock initialized: now 1790348270013469 us; error 26 us; skew 500 ppm
Sep 25 14:57:50 dist-test-slave-8qt6 krb5kdc[10465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348270, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:50.016237 10478 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:50.016739 10478 webserver.cc:460] Webserver started at http://127.20.199.62:45163/ using document root <none> and password file <none>
I20260925 14:57:50.016975 10478 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:50.017024 10478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:50.017204 10478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:50.018371 10478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "2320302f25224ef8a4b63f900ed94174"
format_stamp: "Formatted at 2026-09-25 14:57:50 on dist-test-slave-8qt6"
server_key: "b617a6301fcbbd59c0cfd9df833a30b5"
server_key_iv: "e3c212a286ef54a3419f5c004b52cb50"
server_key_version: "encryptionkey@0"
I20260925 14:57:50.018762 10478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "2320302f25224ef8a4b63f900ed94174"
format_stamp: "Formatted at 2026-09-25 14:57:50 on dist-test-slave-8qt6"
server_key: "b617a6301fcbbd59c0cfd9df833a30b5"
server_key_iv: "e3c212a286ef54a3419f5c004b52cb50"
server_key_version: "encryptionkey@0"
I20260925 14:57:50.020231 10478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:50.021036 10493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:50.021245 10478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:50.021332 10478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "2320302f25224ef8a4b63f900ed94174"
format_stamp: "Formatted at 2026-09-25 14:57:50 on dist-test-slave-8qt6"
server_key: "b617a6301fcbbd59c0cfd9df833a30b5"
server_key_iv: "e3c212a286ef54a3419f5c004b52cb50"
server_key_version: "encryptionkey@0"
I20260925 14:57:50.021385 10478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:50.036232 10478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:50.038795 10478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:50.038920 10478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:50.042868 10478 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:41475
I20260925 14:57:50.042935 10545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:41475 every 8 connection(s)
I20260925 14:57:50.043270 10478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:50.044327 10546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:50.046782 10546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174: Bootstrap starting.
I20260925 14:57:50.047389 10546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:50.047709 10546 log.cc:824] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:50.048445 10546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174: No bootstrap required, opened a new log
I20260925 14:57:50.050205 10546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2320302f25224ef8a4b63f900ed94174" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41475 } }
I20260925 14:57:50.050410 10546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:50.050464 10546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2320302f25224ef8a4b63f900ed94174, State: Initialized, Role: FOLLOWER
I20260925 14:57:50.050642 10546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2320302f25224ef8a4b63f900ed94174" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41475 } }
I20260925 14:57:50.050748 10546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:50.050792 10546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:50.050849 10546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:50.051683 10546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2320302f25224ef8a4b63f900ed94174" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41475 } }
I20260925 14:57:50.051805 10546 leader_election.cc:302] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2320302f25224ef8a4b63f900ed94174; no voters:
I20260925 14:57:50.052011 10546 leader_election.cc:288] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:50.052101 10549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:50.052253 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10478
I20260925 14:57:50.052282 10546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:50.052348 10549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [term 1 LEADER]: Becoming Leader. State: Replica: 2320302f25224ef8a4b63f900ed94174, State: Running, Role: LEADER
I20260925 14:57:50.052386 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:50.052538 10549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2320302f25224ef8a4b63f900ed94174" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41475 } }
I20260925 14:57:50.052616 21276 external_mini_cluster.cc:1428] Setting key 9c3d8c1a35e19773eae5f3f5a9101a9f
I20260925 14:57:50.052959 10550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2320302f25224ef8a4b63f900ed94174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2320302f25224ef8a4b63f900ed94174" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41475 } } }
I20260925 14:57:50.053032 10550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:50.053190 10551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2320302f25224ef8a4b63f900ed94174. Latest consensus state: current_term: 1 leader_uuid: "2320302f25224ef8a4b63f900ed94174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2320302f25224ef8a4b63f900ed94174" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 41475 } } }
I20260925 14:57:50.053256 10551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:50.053793 10559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:50.054467 10559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:50.056144 10559 catalog_manager.cc:1347] Generated new cluster ID: 7f2e6c29e6b145d1aec8f491a6c1270d
I20260925 14:57:50.056202 10559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:50.059230 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.053267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55693 (local address 127.20.199.62:41475)
0925 14:57:50.053926 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.053937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.053973 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:57:50.054192 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.054199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.054243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.054395 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:50.054618 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.055838 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.056954 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.057624 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.057753 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.058510 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:50.058514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:50.058514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:50.058731 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:50.058774 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:50.058790 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:50.058892 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.058994 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.059023 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":63,"threads_started":1}
I20260925 14:57:50.060492 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:50.061043 10559 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:50.061829 10559 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:50.067178 10559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2320302f25224ef8a4b63f900ed94174: Generated new TSK 0
I20260925 14:57:50.067343 10559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:50.070679 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.065805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40463 (local address 127.20.199.62:41475)
0925 14:57:50.066760 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:50.066762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:50.067522 (+ 760us) server_negotiation.cc:407] Connection header received
0925 14:57:50.067604 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:50.067606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:50.067616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:50.067672 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:50.067775 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.068577 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.069299 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:50.069669 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:50.069744 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:50.070422 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:50.070423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:50.070423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:50.070428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:50.070443 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:50.070447 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:50.070511 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:50.070580 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:57:50.070594 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934}
I20260925 14:57:50.072827 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10478
W20260925 14:57:50.076701 10463 connection.cc:535] client connection to 127.20.199.62:41475 recv error: Network error: failed to read from TLS socket (remote: 127.20.199.62:41475): Connection reset by peer (error 104)
2026-09-25T14:57:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:50 dist-test-slave-8qt6 krb5kdc[10577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:50 dist-test-slave-8qt6 krb5kdc[10577](info): set up 2 sockets
Sep 25 14:57:50 dist-test-slave-8qt6 krb5kdc[10577](info): commencing operation
krb5kdc: starting...
W20260925 14:57:51.117600 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:51 dist-test-slave-8qt6 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:51.233981 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:44353
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:51.313575 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:51.313768 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:51.313791 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:51.314842 10590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:51.314877 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:51.314890 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:51.314903 10590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:51.314914 10590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:51.316256 10590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:44353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:51.316433 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:51.316599 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:51.321251 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:51.321251 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:51.321274 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:51.321875 10590 server_base.cc:1034] running on GCE node
I20260925 14:57:51.322125 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:51.322364 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:51.323511 10590 hybrid_clock.cc:648] HybridClock initialized: now 1790348271323496 us; error 30 us; skew 500 ppm
Sep 25 14:57:51 dist-test-slave-8qt6 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348271, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:51.332409 10590 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:51.332800 10590 webserver.cc:460] Webserver started at http://127.20.199.62:38859/ using document root <none> and password file <none>
I20260925 14:57:51.333029 10590 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:51.333070 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:51.333156 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:51.334213 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "2854e94968b54e16b2a4cf86ae0fe8a1"
format_stamp: "Formatted at 2026-09-25 14:57:51 on dist-test-slave-8qt6"
server_key: "bd67648c336beca6a8489789ab73a154"
server_key_iv: "d124251d77ec46f5a1e57cd955e5b29c"
server_key_version: "encryptionkey@0"
I20260925 14:57:51.334506 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "2854e94968b54e16b2a4cf86ae0fe8a1"
format_stamp: "Formatted at 2026-09-25 14:57:51 on dist-test-slave-8qt6"
server_key: "bd67648c336beca6a8489789ab73a154"
server_key_iv: "d124251d77ec46f5a1e57cd955e5b29c"
server_key_version: "encryptionkey@0"
I20260925 14:57:51.335799 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:57:51.336499 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:51.336679 10590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:57:51.336784 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "2854e94968b54e16b2a4cf86ae0fe8a1"
format_stamp: "Formatted at 2026-09-25 14:57:51 on dist-test-slave-8qt6"
server_key: "bd67648c336beca6a8489789ab73a154"
server_key_iv: "d124251d77ec46f5a1e57cd955e5b29c"
server_key_version: "encryptionkey@0"
I20260925 14:57:51.336865 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:51.340199 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:51.342439 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:51.342590 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:51.346311 10590 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:44353
I20260925 14:57:51.346370 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:44353 every 8 connection(s)
I20260925 14:57:51.346689 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:51.347637 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:51.348385 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10590
I20260925 14:57:51.348505 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:51.348693 21276 external_mini_cluster.cc:1428] Setting key 974d4ea61941c68c8262bda381598b7e
I20260925 14:57:51.350191 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1: Bootstrap starting.
I20260925 14:57:51.350895 10658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:51.351223 10658 log.cc:824] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:51.351413 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:51.349217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34857 (local address 127.20.199.62:44353)
0925 14:57:51.349431 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:51.349441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:51.349475 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:57:51.350016 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:51.350023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:51.350049 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:51.350169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:51.350869 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:51.350873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:51.350873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:51.350877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:51.350923 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:51.350966 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:51.351066 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:51.351183 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 14:57:51.351226 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260925 14:57:51.351960 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1: No bootstrap required, opened a new log
I20260925 14:57:51.352438 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:51.354163 10658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44353 } }
I20260925 14:57:51.354314 10658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:51.354360 10658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2854e94968b54e16b2a4cf86ae0fe8a1, State: Initialized, Role: FOLLOWER
I20260925 14:57:51.354483 10658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44353 } }
I20260925 14:57:51.354581 10658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:51.354625 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:51.354673 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:51.355195 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44353 } }
I20260925 14:57:51.355311 10658 leader_election.cc:302] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2854e94968b54e16b2a4cf86ae0fe8a1; no voters:
I20260925 14:57:51.355495 10658 leader_election.cc:288] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:51.355541 10668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:51.355729 10668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [term 1 LEADER]: Becoming Leader. State: Replica: 2854e94968b54e16b2a4cf86ae0fe8a1, State: Running, Role: LEADER
I20260925 14:57:51.355746 10658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:51.357024 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:51.353797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60539 (local address 127.20.199.62:44353)
0925 14:57:51.353830 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:51.353832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:51.355931 (+ 2099us) server_negotiation.cc:407] Connection header received
0925 14:57:51.355993 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:51.355994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:51.355999 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:51.356059 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:51.356745 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:51.356746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:51.356746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:51.356750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:51.356764 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:51.356768 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:51.356809 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:51.356939 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:57:51.356953 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:51.357338 10668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44353 } }
I20260925 14:57:51.358008 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44353 } } }
I20260925 14:57:51.358031 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2854e94968b54e16b2a4cf86ae0fe8a1. Latest consensus state: current_term: 1 leader_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2854e94968b54e16b2a4cf86ae0fe8a1" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 44353 } } }
I20260925 14:57:51.358264 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:51.358271 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:51.358693 10682 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:51.358755 10682 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:51.358822 10684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:51.358989 10684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:51.360303 10684 catalog_manager.cc:1347] Generated new cluster ID: fb3fb1f2858d458abae6cdeb8e0bca08
I20260925 14:57:51.360353 10684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:51.365648 10684 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:51.365814 10684 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:51.369282 10684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2854e94968b54e16b2a4cf86ae0fe8a1: Generated new TSK 0
I20260925 14:57:51.369480 10684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:51.371383 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10590
2026-09-25T14:57:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:51 dist-test-slave-8qt6 krb5kdc[10689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:51 dist-test-slave-8qt6 krb5kdc[10689](info): set up 2 sockets
Sep 25 14:57:51 dist-test-slave-8qt6 krb5kdc[10689](info): commencing operation
krb5kdc: starting...
W20260925 14:57:52.413553 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:52 dist-test-slave-8qt6 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:52.532936 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:43981
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:52.615113 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:52.615351 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:52.615388 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:52.616535 10702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:52.616602 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:52.616629 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:52.616660 10702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:52.616686 10702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:52.618250 10702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:43981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:52.618532 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:52.618760 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:52.623713 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:52.623747 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:52.623747 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:52.623951 10702 server_base.cc:1034] running on GCE node
I20260925 14:57:52.624224 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:52.624509 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:52.625684 10702 hybrid_clock.cc:648] HybridClock initialized: now 1790348272625664 us; error 41 us; skew 500 ppm
Sep 25 14:57:52 dist-test-slave-8qt6 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:52.628372 10702 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:52.628897 10702 webserver.cc:460] Webserver started at http://127.20.199.62:33115/ using document root <none> and password file <none>
I20260925 14:57:52.629124 10702 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:52.629169 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:52.629328 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:52.630475 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "c69622ddf35549b48024dc5b4ea65b72"
format_stamp: "Formatted at 2026-09-25 14:57:52 on dist-test-slave-8qt6"
server_key: "181ea0908968bedd06db2b24a8f3981d"
server_key_iv: "d929ee1402f54d6ef1743ddb4cfd0a97"
server_key_version: "encryptionkey@0"
I20260925 14:57:52.630836 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "c69622ddf35549b48024dc5b4ea65b72"
format_stamp: "Formatted at 2026-09-25 14:57:52 on dist-test-slave-8qt6"
server_key: "181ea0908968bedd06db2b24a8f3981d"
server_key_iv: "d929ee1402f54d6ef1743ddb4cfd0a97"
server_key_version: "encryptionkey@0"
I20260925 14:57:52.632184 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:52.632916 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:52.633104 10702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:52.633178 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "c69622ddf35549b48024dc5b4ea65b72"
format_stamp: "Formatted at 2026-09-25 14:57:52 on dist-test-slave-8qt6"
server_key: "181ea0908968bedd06db2b24a8f3981d"
server_key_iv: "d929ee1402f54d6ef1743ddb4cfd0a97"
server_key_version: "encryptionkey@0"
I20260925 14:57:52.633266 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:52.653132 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:52.655560 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:52.655736 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:52.659760 10702 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:43981
I20260925 14:57:52.659818 10769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:43981 every 8 connection(s)
I20260925 14:57:52.660176 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:52.661018 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:52.663249 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72: Bootstrap starting.
I20260925 14:57:52.663767 10770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:52.663995 10770 log.cc:824] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:52.664659 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72: No bootstrap required, opened a new log
I20260925 14:57:52.666278 10770 raft_consensus.cc:348] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69622ddf35549b48024dc5b4ea65b72" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43981 } }
I20260925 14:57:52.666414 10770 raft_consensus.cc:374] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:52.666435 10770 raft_consensus.cc:729] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69622ddf35549b48024dc5b4ea65b72, State: Initialized, Role: FOLLOWER
I20260925 14:57:52.666536 10770 consensus_queue.cc:260] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69622ddf35549b48024dc5b4ea65b72" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43981 } }
I20260925 14:57:52.666618 10770 raft_consensus.cc:388] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:52.666654 10770 raft_consensus.cc:482] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:52.666723 10770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:52.667593 10770 raft_consensus.cc:504] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69622ddf35549b48024dc5b4ea65b72" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43981 } }
I20260925 14:57:52.667719 10770 leader_election.cc:302] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c69622ddf35549b48024dc5b4ea65b72; no voters:
I20260925 14:57:52.667724 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10702
I20260925 14:57:52.667834 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:52.667891 10770 leader_election.cc:288] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:52.667936 10773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:52.668084 21276 external_mini_cluster.cc:1428] Setting key 32348abaa34294f72cf1010e82d9b237
I20260925 14:57:52.668107 10773 raft_consensus.cc:686] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [term 1 LEADER]: Becoming Leader. State: Replica: c69622ddf35549b48024dc5b4ea65b72, State: Running, Role: LEADER
I20260925 14:57:52.668202 10770 sys_catalog.cc:564] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:52.668183 10773 consensus_queue.cc:237] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69622ddf35549b48024dc5b4ea65b72" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43981 } }
I20260925 14:57:52.668586 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c69622ddf35549b48024dc5b4ea65b72. Latest consensus state: current_term: 1 leader_uuid: "c69622ddf35549b48024dc5b4ea65b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69622ddf35549b48024dc5b4ea65b72" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43981 } } }
I20260925 14:57:52.668668 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:52.668823 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c69622ddf35549b48024dc5b4ea65b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69622ddf35549b48024dc5b4ea65b72" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 43981 } } }
I20260925 14:57:52.668874 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:52.669272 10781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:52.669880 10781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:52.671514 10781 catalog_manager.cc:1347] Generated new cluster ID: 52336df998da454385cc3a4360d8199d
I20260925 14:57:52.671584 10781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:57:52 dist-test-slave-8qt6 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:52.679955 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:52.668830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34999 (local address 127.20.199.62:43981)
0925 14:57:52.669008 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:52.669018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:52.669280 (+ 262us) server_negotiation.cc:407] Connection header received
0925 14:57:52.669436 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:52.669446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:52.669508 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:52.669683 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:52.670464 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:52.671816 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:52.672860 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:52.673497 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:52.673609 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:52.676415 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:52.676420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:52.676421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:52.676432 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:52.678335 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:52.678738 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:52.678741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:52.678742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:52.678854 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:52.679077 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:52.679078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:52.679078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:52.679357 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:52.679491 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:52.679628 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:57:52.679755 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260925 14:57:52.680951 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:52.684751 10781 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:52.685449 10781 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:52.689599 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:52.682370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41407 (local address 127.20.199.62:43981)
0925 14:57:52.682405 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:52.682406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:52.682527 (+ 121us) server_negotiation.cc:407] Connection header received
0925 14:57:52.682583 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:52.682583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:52.682594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:52.682643 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:52.683239 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:52.683981 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:52.684982 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:52.685284 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:52.685355 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:52.686643 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:52.686644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:52.686645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:52.686651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:52.688185 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:52.688547 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:52.688548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:52.688548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:52.688606 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:52.688806 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:52.688807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:52.688807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:52.689019 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:52.689108 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:52.689347 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 14:57:52.689509 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:52.689693 10781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c69622ddf35549b48024dc5b4ea65b72: Generated new TSK 0
I20260925 14:57:52.689900 10781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:52.695173 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10702
2026-09-25T14:57:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:52 dist-test-slave-8qt6 krb5kdc[10801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:52 dist-test-slave-8qt6 krb5kdc[10801](info): set up 2 sockets
Sep 25 14:57:52 dist-test-slave-8qt6 krb5kdc[10801](info): commencing operation
krb5kdc: starting...
W20260925 14:57:53.733582 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:53 dist-test-slave-8qt6 krb5kdc[10801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:53.852298 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42351
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:53.931269 10814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:53.931483 10814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:53.931507 10814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:53.933069 10814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:53.933145 10814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:53.933169 10814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:53.933187 10814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:53.933207 10814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:53.935863 10814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:42351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:53.936329 10814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:53.936628 10814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:53.944295 10822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:53.944293 10820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:53.944330 10819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:53.944796 10814 server_base.cc:1034] running on GCE node
I20260925 14:57:53.945065 10814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:53.945477 10814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:53.946674 10814 hybrid_clock.cc:648] HybridClock initialized: now 1790348273946644 us; error 47 us; skew 500 ppm
Sep 25 14:57:53 dist-test-slave-8qt6 krb5kdc[10801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348273, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:53.949116 10814 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:53.949647 10814 webserver.cc:460] Webserver started at http://127.20.199.62:46529/ using document root <none> and password file <none>
I20260925 14:57:53.949908 10814 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:53.949975 10814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:53.950134 10814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:53.951196 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "fb322143e874413fbdd02a852ae8c876"
format_stamp: "Formatted at 2026-09-25 14:57:53 on dist-test-slave-8qt6"
server_key: "73508174176ec9d5ac651209b72a589b"
server_key_iv: "7d04cd3f009a7d9cb0667610abe74d34"
server_key_version: "encryptionkey@0"
I20260925 14:57:53.951570 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "fb322143e874413fbdd02a852ae8c876"
format_stamp: "Formatted at 2026-09-25 14:57:53 on dist-test-slave-8qt6"
server_key: "73508174176ec9d5ac651209b72a589b"
server_key_iv: "7d04cd3f009a7d9cb0667610abe74d34"
server_key_version: "encryptionkey@0"
I20260925 14:57:53.952885 10814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:57:53.953660 10829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:53.953827 10814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:53.953934 10814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "fb322143e874413fbdd02a852ae8c876"
format_stamp: "Formatted at 2026-09-25 14:57:53 on dist-test-slave-8qt6"
server_key: "73508174176ec9d5ac651209b72a589b"
server_key_iv: "7d04cd3f009a7d9cb0667610abe74d34"
server_key_version: "encryptionkey@0"
I20260925 14:57:53.954010 10814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:53.962729 10814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:53.965202 10814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:53.965443 10814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:53.969870 10814 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:42351
I20260925 14:57:53.969944 10881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:42351 every 8 connection(s)
I20260925 14:57:53.970304 10814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:53.971203 10882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:53.973394 10882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876: Bootstrap starting.
I20260925 14:57:53.974028 10882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:53.974261 10882 log.cc:824] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:53.974911 10882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876: No bootstrap required, opened a new log
I20260925 14:57:53.976442 10882 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb322143e874413fbdd02a852ae8c876" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42351 } }
I20260925 14:57:53.976603 10882 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:53.976629 10882 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb322143e874413fbdd02a852ae8c876, State: Initialized, Role: FOLLOWER
I20260925 14:57:53.976732 10882 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb322143e874413fbdd02a852ae8c876" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42351 } }
I20260925 14:57:53.976824 10882 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:53.976861 10882 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:53.976914 10882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:53.976984 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10814
I20260925 14:57:53.977104 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:53.977299 21276 external_mini_cluster.cc:1428] Setting key 597aab5e3d44e3ff864f38239d0072b1
I20260925 14:57:53.978150 10882 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb322143e874413fbdd02a852ae8c876" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42351 } }
I20260925 14:57:53.978245 10882 leader_election.cc:302] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb322143e874413fbdd02a852ae8c876; no voters:
I20260925 14:57:53.978461 10882 leader_election.cc:288] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:53.978506 10887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:53.978689 10887 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [term 1 LEADER]: Becoming Leader. State: Replica: fb322143e874413fbdd02a852ae8c876, State: Running, Role: LEADER
I20260925 14:57:53.978811 10882 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:53.978929 10887 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb322143e874413fbdd02a852ae8c876" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42351 } }
I20260925 14:57:53.980243 10889 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb322143e874413fbdd02a852ae8c876. Latest consensus state: current_term: 1 leader_uuid: "fb322143e874413fbdd02a852ae8c876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb322143e874413fbdd02a852ae8c876" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42351 } } }
I20260925 14:57:53.980336 10889 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:53.980602 10888 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb322143e874413fbdd02a852ae8c876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb322143e874413fbdd02a852ae8c876" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 42351 } } }
I20260925 14:57:53.980667 10888 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876 [sys.catalog]: This master's current role is: LEADER
W20260925 14:57:53.980813 10902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:53.980857 10902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:53.980924 10904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:53.981094 10904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:53.982949 10904 catalog_manager.cc:1347] Generated new cluster ID: edd104907c7f40988796ef5ff468c052
I20260925 14:57:53.983008 10904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:53.983414 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:53.977836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38183 (local address 127.20.199.62:42351)
0925 14:57:53.978082 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:53.978092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:53.978131 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:57:53.978251 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:53.978259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:53.978315 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:53.978475 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:53.978708 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:53.979968 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:53.981295 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:53.981884 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:53.982002 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:53.982685 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:53.982689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:53.982689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:53.982903 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:53.982941 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:53.982956 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:53.983027 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:53.983162 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 14:57:53.983224 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":157,"threads_started":1}
I20260925 14:57:53.984359 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:53.989112 10904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:53.989735 10904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:53.994165 10904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb322143e874413fbdd02a852ae8c876: Generated new TSK 0
I20260925 14:57:53.994355 10904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:53.996455 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:53.985784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42345 (local address 127.20.199.62:42351)
0925 14:57:53.985815 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:53.985817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:53.988795 (+ 2978us) server_negotiation.cc:407] Connection header received
0925 14:57:53.988868 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:53.988869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:53.988890 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:53.988958 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:53.993540 (+ 4582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:53.994330 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:53.995099 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:53.995423 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:53.995502 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:53.996170 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:53.996171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:53.996171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:53.996178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:53.996193 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:53.996197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:53.996264 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:53.996325 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:57:53.996370 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:57:53.998762 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10814
2026-09-25T14:57:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:54 dist-test-slave-8qt6 krb5kdc[10913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:54 dist-test-slave-8qt6 krb5kdc[10913](info): set up 2 sockets
Sep 25 14:57:54 dist-test-slave-8qt6 krb5kdc[10913](info): commencing operation
krb5kdc: starting...
W20260925 14:57:55.033530 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:55 dist-test-slave-8qt6 krb5kdc[10913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:55.153071 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40143
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:55.233088 10926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:55.233327 10926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:55.233366 10926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:55.234540 10926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:55.234615 10926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:55.234642 10926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:55.234673 10926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:55.234696 10926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:55.236156 10926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:55.236420 10926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:55.236645 10926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:55.241420 10932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:55.241420 10934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:55.241596 10926 server_base.cc:1034] running on GCE node
W20260925 14:57:55.241492 10931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:55.241919 10926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:55.242157 10926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:55.243305 10926 hybrid_clock.cc:648] HybridClock initialized: now 1790348275243266 us; error 52 us; skew 500 ppm
Sep 25 14:57:55 dist-test-slave-8qt6 krb5kdc[10913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:55.245713 10926 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:55.246181 10926 webserver.cc:460] Webserver started at http://127.20.199.62:37265/ using document root <none> and password file <none>
I20260925 14:57:55.246402 10926 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:55.246466 10926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:55.246619 10926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:55.247656 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "d7bb228f2ff24a0cbd0d19455a3c8421"
format_stamp: "Formatted at 2026-09-25 14:57:55 on dist-test-slave-8qt6"
server_key: "8932e93039884005928f2041b12e6ad7"
server_key_iv: "e16a79ccb2ee24a8ca2960f827ceee6e"
server_key_version: "encryptionkey@0"
I20260925 14:57:55.248018 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "d7bb228f2ff24a0cbd0d19455a3c8421"
format_stamp: "Formatted at 2026-09-25 14:57:55 on dist-test-slave-8qt6"
server_key: "8932e93039884005928f2041b12e6ad7"
server_key_iv: "e16a79ccb2ee24a8ca2960f827ceee6e"
server_key_version: "encryptionkey@0"
I20260925 14:57:55.249336 10926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:55.250185 10941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:55.250386 10926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:55.250485 10926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "d7bb228f2ff24a0cbd0d19455a3c8421"
format_stamp: "Formatted at 2026-09-25 14:57:55 on dist-test-slave-8qt6"
server_key: "8932e93039884005928f2041b12e6ad7"
server_key_iv: "e16a79ccb2ee24a8ca2960f827ceee6e"
server_key_version: "encryptionkey@0"
I20260925 14:57:55.250582 10926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:55.253846 10926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:55.255716 10926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:55.255867 10926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:55.259761 10926 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:40143
I20260925 14:57:55.259789 10993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:40143 every 8 connection(s)
I20260925 14:57:55.260105 10926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:55.261034 10994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:55.263243 10994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421: Bootstrap starting.
I20260925 14:57:55.263811 10994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:55.264035 10994 log.cc:824] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:55.264906 10994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421: No bootstrap required, opened a new log
I20260925 14:57:55.267356 10994 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40143 } }
I20260925 14:57:55.267576 10994 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:55.267608 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 10926
I20260925 14:57:55.267659 10994 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bb228f2ff24a0cbd0d19455a3c8421, State: Initialized, Role: FOLLOWER
I20260925 14:57:55.267722 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:55.267840 10994 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40143 } }
I20260925 14:57:55.267920 10994 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:55.267935 21276 external_mini_cluster.cc:1428] Setting key a318c31a13a26a2fb8a50a6b9b0440fd
I20260925 14:57:55.267941 10994 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:55.267969 10994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:55.269132 10994 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40143 } }
I20260925 14:57:55.269618 10994 leader_election.cc:302] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7bb228f2ff24a0cbd0d19455a3c8421; no voters:
I20260925 14:57:55.269793 10994 leader_election.cc:288] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:55.269836 10999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:55.270020 10999 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [term 1 LEADER]: Becoming Leader. State: Replica: d7bb228f2ff24a0cbd0d19455a3c8421, State: Running, Role: LEADER
I20260925 14:57:55.270071 10994 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:55.270169 10999 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40143 } }
I20260925 14:57:55.270300 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:55.268397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59787 (local address 127.20.199.62:40143)
0925 14:57:55.268622 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:55.268634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:55.268675 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:57:55.268819 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:55.268826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:55.268858 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:55.269008 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:55.269679 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:55.269683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:55.269683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:55.269688 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:55.269741 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:55.269755 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:55.269824 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:55.269999 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:57:55.270057 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
I20260925 14:57:55.270586 10999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7bb228f2ff24a0cbd0d19455a3c8421. Latest consensus state: current_term: 1 leader_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40143 } } }
I20260925 14:57:55.270730 10999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:55.270598 11000 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bb228f2ff24a0cbd0d19455a3c8421" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 40143 } } }
I20260925 14:57:55.271108 11000 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:55.270974 11008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:55.271256 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:55.271741 11008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:55.273442 11008 catalog_manager.cc:1347] Generated new cluster ID: 6a8a25d6aa7a4854888b0b3177bfaf76
I20260925 14:57:55.273504 11008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:55.274511 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:55.272776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58071 (local address 127.20.199.62:40143)
0925 14:57:55.272809 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:55.272811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:55.273596 (+ 785us) server_negotiation.cc:407] Connection header received
0925 14:57:55.273657 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:55.273658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:55.273664 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:55.273726 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:55.274288 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:55.274289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:55.274290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:57:55.274294 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:55.274308 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:57:55.274313 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:55.274376 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:55.274433 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:57:55.274447 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:57:55.284451 11008 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:55.284653 11008 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:55.289032 11008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7bb228f2ff24a0cbd0d19455a3c8421: Generated new TSK 0
I20260925 14:57:55.289227 11008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:55.292266 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 10926
2026-09-25T14:57:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1293 ms)
[----------] 18 tests from AuthTokenIssuingTest (23761 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:55 dist-test-slave-8qt6 krb5kdc[11025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:55 dist-test-slave-8qt6 krb5kdc[11025](info): set up 2 sockets
Sep 25 14:57:55 dist-test-slave-8qt6 krb5kdc[11025](info): commencing operation
krb5kdc: starting...
W20260925 14:57:56.325479 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:56.444167 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45603
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:45603,127.20.199.61:44147,127.20.199.60:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:56.524489 11038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:56.524708 11038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:56.524724 11038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:56.524744 11038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:56.525916 11038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:56.525955 11038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:56.525969 11038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:56.525981 11038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:56.525993 11038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:56.527477 11038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:45603,127.20.199.61:44147,127.20.199.60:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:45603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:56.527726 11038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:56.527948 11038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:56.532773 11043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:56.532750 11044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:56.532775 11046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:56.533175 11038 server_base.cc:1034] running on GCE node
I20260925 14:57:56.533476 11038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:56.533744 11038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:56.534910 11038 hybrid_clock.cc:648] HybridClock initialized: now 1790348276534890 us; error 34 us; skew 500 ppm
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:56.544056 11038 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:56.544499 11038 webserver.cc:460] Webserver started at http://127.20.199.62:42671/ using document root <none> and password file <none>
I20260925 14:57:56.544700 11038 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:56.544744 11038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:56.544840 11038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:56.545979 11038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "1ea3251d06424bd09078045a36308649"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "1607ed73dd6ac971740015c635b2e94b"
server_key_iv: "a32e053ed1e6e627c3573402fa327c7a"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.546299 11038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "1ea3251d06424bd09078045a36308649"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "1607ed73dd6ac971740015c635b2e94b"
server_key_iv: "a32e053ed1e6e627c3573402fa327c7a"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.547700 11038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:56.548425 11053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:56.548678 11038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:56.548792 11038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "1ea3251d06424bd09078045a36308649"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "1607ed73dd6ac971740015c635b2e94b"
server_key_iv: "a32e053ed1e6e627c3573402fa327c7a"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.548882 11038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:56.555692 11038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:56.557478 11038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:56.557631 11038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:56.561467 11038 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:45603
I20260925 14:57:56.561493 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:45603 every 8 connection(s)
I20260925 14:57:56.561869 11038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:56.562790 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:56.564294 11106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.569068 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 11038
I20260925 14:57:56.569190 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:56.569372 21276 external_mini_cluster.cc:1428] Setting key 3c2dc759f740e35b5e2a3fec1f98c361
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:56.574239 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.564987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.20.199.62:45603)
0925 14:57:56.565186 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.565195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.565253 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:57:56.565811 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.565813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.565829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.565906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.566398 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.567187 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.567989 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.568288 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.568363 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.570982 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.570991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.570991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.570999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.572591 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.573059 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.573061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.573061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.573135 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.573456 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.573456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.573457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.573662 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.573728 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.573978 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.574083 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260925 14:57:56.574379 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.564960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45603 (local address 127.0.0.1:35024)
0925 14:57:56.565154 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.565186 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.565685 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.565932 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.565934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.566173 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.566321 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.566324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.567266 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.567266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.567920 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.567921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.568383 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.568384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.568437 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.568972 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.568981 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.570864 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.572678 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.572685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.572687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.572968 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.573228 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.573229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.573229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.573349 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.573738 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.573744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.573828 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.573926 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.574198 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":91,"thread_start_us":111,"threads_started":1}
I20260925 14:57:56.575136 11106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.575515 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.575376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55950)
0925 14:57:56.575427 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.575482 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:57:56.575666 11057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
I20260925 14:57:56.576105 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.575992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55960)
0925 14:57:56.576048 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.576080 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:57:56.576280 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:44147: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.20.199.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab.
Entry for principal kudu/127.20.199.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab.
I20260925 14:57:56.596405 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:44147
--webserver_interface=127.20.199.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab
--principal=kudu/127.20.199.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:45603,127.20.199.61:44147,127.20.199.60:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:56.625502 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } attempt: 1
I20260925 14:57:56.626040 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.625886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55966)
0925 14:57:56.625952 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.625998 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:57:56.626425 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.626327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55974)
0925 14:57:56.626373 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.626402 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:57:56.626524 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:44147: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
W20260925 14:57:56.677459 11113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:56.677675 11113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:56.677693 11113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:56.677713 11113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:56.678807 11113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:56.678848 11113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:56.678862 11113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:56.678875 11113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:56.678887 11113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:56.680317 11113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:45603,127.20.199.61:44147,127.20.199.60:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab
--principal=kudu/127.20.199.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.199.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:56.680552 11113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:56.680783 11113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:56.685726 11118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:56.685700 11121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:56.685735 11113 server_base.cc:1034] running on GCE node
W20260925 14:57:56.685717 11119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:56.686242 11113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:56.686559 11113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:56.687741 11113 hybrid_clock.cc:648] HybridClock initialized: now 1790348276687733 us; error 35 us; skew 500 ppm
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:56.690412 11113 init.cc:377] Logged in from keytab as kudu/127.20.199.61@KRBTEST.COM (short username kudu)
I20260925 14:57:56.690919 11113 webserver.cc:460] Webserver started at http://127.20.199.61:44191/ using document root <none> and password file <none>
I20260925 14:57:56.691123 11113 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:56.691164 11113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:56.691250 11113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:56.692268 11113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data/instance:
uuid: "11fa1bcda9084bbd9e595704d6910989"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "ac640b2248d7f0d5a884905e8cec99be"
server_key_iv: "69d30f4bc932bd94f193f5a1712b354f"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.692576 11113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal/instance:
uuid: "11fa1bcda9084bbd9e595704d6910989"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "ac640b2248d7f0d5a884905e8cec99be"
server_key_iv: "69d30f4bc932bd94f193f5a1712b354f"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.693941 11113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:56.694630 11128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:56.694689 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } attempt: 2
I20260925 14:57:56.694823 11113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:56.694933 11113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal
uuid: "11fa1bcda9084bbd9e595704d6910989"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "ac640b2248d7f0d5a884905e8cec99be"
server_key_iv: "69d30f4bc932bd94f193f5a1712b354f"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.695030 11113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:56.695102 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.694972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55980)
0925 14:57:56.695035 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.695067 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:57:56.695408 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.695315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55982)
0925 14:57:56.695361 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.695385 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:57:56.695506 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:44147: Network error: Client connection negotiation failed: client connection to 127.20.199.61:44147: connect: Connection refused (error 111)
I20260925 14:57:56.708727 11113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:56.710868 11113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:56.711058 11113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:56.714923 11113 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.61:44147
I20260925 14:57:56.714982 11180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.61:44147 every 8 connection(s)
I20260925 14:57:56.715340 11113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
I20260925 14:57:56.716439 11181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:56.718073 11181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.721227 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 11113
I20260925 14:57:56.721307 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-1/wal/instance
I20260925 14:57:56.721514 21276 external_mini_cluster.cc:1428] Setting key 864e210862fddaff82aeba74a6c6b394
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:56.727360 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.718837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.20.199.62:45603)
0925 14:57:56.718930 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.718933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.719147 (+ 214us) server_negotiation.cc:407] Connection header received
0925 14:57:56.719650 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.719651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.719664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.719730 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.720348 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.720845 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.722043 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.722263 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.722352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.724483 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.724485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.724486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.724494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.725941 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.726359 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.726360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.726360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.726420 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.726679 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.726680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.726681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.726806 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.726883 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.727180 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.727267 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:57:56.727545 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.718793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45603 (local address 127.0.0.1:35036)
0925 14:57:56.719020 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.719057 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.719541 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.719804 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.719807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.720058 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.720247 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.720251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.720965 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.720966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.721955 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.721957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.722375 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.722376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.722507 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.723055 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.723064 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.724384 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.726040 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.726048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.726050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.726272 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.726494 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.726495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.726496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.726604 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.726875 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.726881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.727043 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.727148 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.727305 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260925 14:57:56.728140 11181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.199.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.60@KRBTEST.COM" created.
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:57:56.736034 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.728464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.20.199.61:44147)
0925 14:57:56.728636 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.728641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.728660 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:57:56.728716 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.728718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.728728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.728791 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.729078 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.730295 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.731038 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.731392 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.731489 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.733269 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.733277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.733277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.733285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.734864 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.735241 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.735243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.735244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.735295 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.735500 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.735501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.735501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.735636 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.735696 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.735866 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.735935 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260925 14:57:56.736171 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.728425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:55996)
0925 14:57:56.728482 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.728494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.728553 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.728822 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.728822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.728944 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.728999 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.729000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.730421 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.730422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.730955 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.730957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.731488 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.731489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.731552 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.731996 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.732000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.733188 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.734946 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.734947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.734947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.735164 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.735360 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.735361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.735361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.735435 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.735703 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.735704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.735757 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.735845 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.736003 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":49}
I20260925 14:57:56.736824 11181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.737214 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.737083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56946)
0925 14:57:56.737136 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.737187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:57:56.737358 11132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
I20260925 14:57:56.737874 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.737763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56954)
0925 14:57:56.737814 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.737846 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:57:56.737993 11181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:43703: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab.
Entry for principal kudu/127.20.199.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab.
I20260925 14:57:56.748275 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:43703
--webserver_interface=127.20.199.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab
--principal=kudu/127.20.199.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:45603,127.20.199.61:44147,127.20.199.60:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:56.786643 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } attempt: 3
I20260925 14:57:56.787225 11181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } attempt: 1
I20260925 14:57:56.787632 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.787511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56966)
0925 14:57:56.787570 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.787604 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:57:56.788005 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.787911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56968)
0925 14:57:56.787956 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.787982 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:57:56.788093 11181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:43703: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:57:56.794104 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.787069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:56012)
0925 14:57:56.787162 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.787175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.787246 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.787447 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.787448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.787647 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.787724 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.787727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.788663 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.788664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.789213 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.789214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.789745 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.789746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.789804 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.790254 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.790258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.791460 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.793161 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.793161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.793162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.793329 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.793566 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.793567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.793567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.793621 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.793871 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.793872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.793910 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.793961 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.794038 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 14:57:56.794306 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.787143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.20.199.61:44147)
0925 14:57:56.787207 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.787210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.787229 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:57:56.787318 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.787319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.787333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.787393 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.787811 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.788579 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.789295 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.789674 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.789746 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.791550 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.791552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.791552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.791560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.793081 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.793427 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.793428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.793428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.793492 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.793682 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.793683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.793683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.793813 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.793875 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.794008 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.794099 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:57:56.794750 11106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.795085 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.794976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56970)
0925 14:57:56.795024 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.795058 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:57:56.795495 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.795407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56978)
0925 14:57:56.795451 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.795476 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:57:56.795590 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:43703: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
I20260925 14:57:56.826764 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } attempt: 1
I20260925 14:57:56.827293 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.827132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:56990)
0925 14:57:56.827203 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.827255 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:57:56.827699 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.827593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:57000)
0925 14:57:56.827649 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.827678 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:57:56.827817 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:43703: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
W20260925 14:57:56.832595 11188 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:56.832794 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:56.832811 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:56.832878 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:56.834092 11188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:56.834136 11188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:56.834151 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:56.834196 11188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:56.834219 11188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:56.835716 11188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:45603,127.20.199.61:44147,127.20.199.60:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab
--principal=kudu/127.20.199.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.199.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:56.835959 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:56.836225 11188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:56.841094 11188 server_base.cc:1034] running on GCE node
W20260925 14:57:56.841075 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:56.841063 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:56.841075 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:56.841677 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:56.841972 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:56.843133 11188 hybrid_clock.cc:648] HybridClock initialized: now 1790348276843115 us; error 30 us; skew 500 ppm
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:56.845487 11188 init.cc:377] Logged in from keytab as kudu/127.20.199.60@KRBTEST.COM (short username kudu)
I20260925 14:57:56.845950 11188 webserver.cc:460] Webserver started at http://127.20.199.60:42655/ using document root <none> and password file <none>
I20260925 14:57:56.846194 11188 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:56.846262 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:56.846403 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:56.847443 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data/instance:
uuid: "bb4f0fbd700f448d8870c5e2a317825c"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "9665866a3ef7fe4c56def68f6a534f80"
server_key_iv: "0b52a667406f1d57cf6e53ddb8fb91ac"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.847793 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal/instance:
uuid: "bb4f0fbd700f448d8870c5e2a317825c"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "9665866a3ef7fe4c56def68f6a534f80"
server_key_iv: "0b52a667406f1d57cf6e53ddb8fb91ac"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.849102 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:56.849872 11203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:56.850028 11188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:56.850138 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal
uuid: "bb4f0fbd700f448d8870c5e2a317825c"
format_stamp: "Formatted at 2026-09-25 14:57:56 on dist-test-slave-8qt6"
server_key: "9665866a3ef7fe4c56def68f6a534f80"
server_key_iv: "0b52a667406f1d57cf6e53ddb8fb91ac"
server_key_version: "encryptionkey@0"
I20260925 14:57:56.850219 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:56.856222 11181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } attempt: 2
I20260925 14:57:56.856609 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.856484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:57012)
0925 14:57:56.856544 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.856582 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:57:56.856920 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.856819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:57018)
0925 14:57:56.856868 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.856891 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:57:56.857002 11181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:43703: Network error: Client connection negotiation failed: client connection to 127.20.199.60:43703: connect: Connection refused (error 111)
I20260925 14:57:56.862103 11188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:56.863878 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:56.864037 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:56.868122 11188 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.60:43703
I20260925 14:57:56.868156 11255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.60:43703 every 8 connection(s)
I20260925 14:57:56.868499 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
I20260925 14:57:56.869539 11256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:56.871061 11256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.872926 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 11188
I20260925 14:57:56.872999 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348014035135-21276-0/minicluster-data/master-2/wal/instance
I20260925 14:57:56.873180 21276 external_mini_cluster.cc:1428] Setting key bc4fac4014ddd4667cf4dca5407965aa
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:56.880983 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.871746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.20.199.62:45603)
0925 14:57:56.871813 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.871815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.872000 (+ 185us) server_negotiation.cc:407] Connection header received
0925 14:57:56.872440 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.872441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.872450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.872525 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.873144 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.873867 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.875088 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.875268 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.875322 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.877969 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.877970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.877970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.877977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.879544 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.879977 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.879978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.879978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.880033 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.880318 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.880319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.880319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.880463 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.880531 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.880855 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.880917 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:57:56.881208 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.871707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:45603 (local address 127.0.0.1:35050)
0925 14:57:56.871870 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.871909 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.872361 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.872578 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.872581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.872813 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.873035 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.873059 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.873942 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.873943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.874988 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.874991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.875359 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.875360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.875490 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.876062 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.876071 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.877881 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.879609 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.879614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.879616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.879904 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.880100 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.880101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.880101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.880240 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.880532 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.880539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.880711 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.880833 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.880968 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260925 14:57:56.881825 11256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.882856 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.873613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.20.199.60:43703)
0925 14:57:56.873782 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.873786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.873876 (+ 90us) server_negotiation.cc:407] Connection header received
0925 14:57:56.873944 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.873948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.873972 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.874061 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.874607 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.875547 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.876250 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.876705 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.876777 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.879490 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.879496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.879496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.879502 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.881348 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.881727 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.881729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.881729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.881780 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.882247 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.882248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.882248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.882429 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.882587 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.882655 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.882739 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:56.888948 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.882314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.20.199.61:44147)
0925 14:57:56.882359 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.882361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.882369 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:56.882398 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.882399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.882406 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.882472 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.882714 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.883302 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.884060 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.884374 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.884425 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.886310 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.886311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.886311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.886320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.887837 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.888239 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.888240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.888240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.888294 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.888494 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.888495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.888495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.888611 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.888680 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.888807 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.888889 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:56.888963 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.882207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:44147 (local address 127.0.0.1:56028)
0925 14:57:56.882260 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.882271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.882307 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.882476 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.882477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.882610 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.882638 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.882640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.883394 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.883395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.883976 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.883978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.884441 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.884442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.884500 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.884832 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.884836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.886240 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.887920 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.887921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.887922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.888135 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.888363 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.888364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.888364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.888430 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.888679 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.888680 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.888716 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.888769 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.888834 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:57:56.889566 11256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:56.891722 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.883610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.20.199.62:45603)
0925 14:57:56.883646 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.883647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.883655 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:56.883696 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.883697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.883706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.883761 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.884280 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.884926 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.886139 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.886348 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.886418 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.888639 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.888640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.888640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.888647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.890605 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.890956 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.890957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.890958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.891016 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.891232 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.891233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.891234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.891341 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.891511 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.891549 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.891624 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:56.896332 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.889854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:57044)
0925 14:57:56.889884 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.889892 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.889949 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.890237 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.890238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.890367 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.890391 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.890392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.891003 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.891004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.891522 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.891524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.892121 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.892122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.892169 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.892573 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.892577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.893828 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.895404 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.895404 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.895405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.895639 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.895820 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.895821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.895821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.895879 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.896121 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.896121 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.896155 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.896209 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.896272 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 14:57:56.896450 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.889890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.20.199.60:43703)
0925 14:57:56.890141 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.890143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.890149 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:57:56.890163 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.890163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.890172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.890235 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.890460 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.890917 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.891610 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.892065 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.892125 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.893910 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.893911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.893911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.893917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.895323 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.895712 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.895713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.895713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.895758 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.895940 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.895941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.895941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.896043 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.896109 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.896252 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.896323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:57:56.898294 11256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: Bootstrap starting.
I20260925 14:57:56.899351 11256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:56.899698 11256 log.cc:824] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:56.899860 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.892468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.20.199.61:44147)
0925 14:57:56.892508 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.892509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.892518 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:56.892576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.892577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.892587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.892651 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.893126 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.893667 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.894381 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.894594 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.894666 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.897107 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.897108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.897108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.897114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.898611 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.899074 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.899076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.899076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.899133 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.899336 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.899336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.899337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.899473 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.899611 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.899683 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.899739 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:56.900607 11256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: No bootstrap required, opened a new log
I20260925 14:57:56.901386 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:56.902631 11256 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.902752 11256 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4f0fbd700f448d8870c5e2a317825c, State: Initialized, Role: FOLLOWER
I20260925 14:57:56.902832 11256 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.902937 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } attempt: 2
I20260925 14:57:56.903163 11256 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:56.902559 11080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ea3251d06424bd09078045a36308649"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35062
I20260925 14:57:56.904671 11262 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.904891 11262 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:57:56.905064 11273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:56.905120 11273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:56.905782 11188 master_runner.cc:186] Error getting master registration for 127.20.199.62:45603: OK, instance_id { permanent_uuid: "1ea3251d06424bd09078045a36308649" instance_seqno: 1790348276560401 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:56.908949 11188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:56.910808 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.903376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.20.199.60:43703)
0925 14:57:56.903427 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.903428 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.903435 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:56.903454 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.903454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.903463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.903508 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.903873 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.904632 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.905382 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.905680 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.905748 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.907855 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.907857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.907857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.907864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.909598 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.910001 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.910002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.910003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.910084 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.910294 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.910294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.910295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.910447 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.910510 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.910648 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.910718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:56.910831 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.903168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:57058)
0925 14:57:56.903247 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.903257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.903294 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.903556 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.903557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.903731 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.903783 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.903785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.904726 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.904727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.905308 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.905310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.905800 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.905802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.905859 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.906260 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.906265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.907769 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.909675 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.909676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.909676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.909897 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.910155 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.910156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.910156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.910225 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.910504 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.910506 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.910554 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.910630 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.910729 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:57:56.912352 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649: Bootstrap starting.
I20260925 14:57:56.913077 11106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:56.913352 11106 log.cc:824] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:56.914299 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649: No bootstrap required, opened a new log
I20260925 14:57:56.915292 11106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.915431 11106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea3251d06424bd09078045a36308649, State: Initialized, Role: FOLLOWER
I20260925 14:57:56.915518 11106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.915791 11276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.915856 11106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:56.915867 11276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:56.915939 11080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:57:56.916062 11080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:56.916857 11080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.917562 11080 leader_election.cc:288] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [CANDIDATE]: Term 1 election: Requested vote from peers 11fa1bcda9084bbd9e595704d6910989 (127.20.199.61:44147), bb4f0fbd700f448d8870c5e2a317825c (127.20.199.60:43703)
W20260925 14:57:56.917886 11287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:56.918004 11287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:56.918494 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ea3251d06424bd09078045a36308649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb4f0fbd700f448d8870c5e2a317825c"
I20260925 14:57:56.918650 11230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:56.919539 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea3251d06424bd09078045a36308649 in term 1.
I20260925 14:57:56.919782 11057 leader_election.cc:302] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea3251d06424bd09078045a36308649, bb4f0fbd700f448d8870c5e2a317825c; no voters:
I20260925 14:57:56.919914 11276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:56.920053 11276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea3251d06424bd09078045a36308649, State: Running, Role: LEADER
I20260925 14:57:56.920140 11276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.920598 11276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea3251d06424bd09078045a36308649. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.920696 11276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:56.921180 11297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:56.918789 11038 master_runner.cc:186] Error getting master registration for 127.20.199.61:44147: OK, instance_id { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" instance_seqno: 1790348276713815 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:56.921548 11038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:56.922021 11297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:56.923834 11230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [term 1 FOLLOWER]: Refusing update from remote peer 1ea3251d06424bd09078045a36308649: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:56.924202 11293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:56.925657 11262 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea3251d06424bd09078045a36308649. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.925750 11262 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:56.934893 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.920120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.20.199.61:44147)
0925 14:57:56.920163 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.920165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.922839 (+ 2674us) server_negotiation.cc:407] Connection header received
0925 14:57:56.923937 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.923938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.923950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.923980 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.925501 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.926193 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.928155 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.928473 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.928549 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.931013 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.931033 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.931034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.931041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.932530 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.933323 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.933325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.933326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.933390 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.934053 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.934055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.934055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.934200 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.934299 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.934669 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.934767 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:56.935487 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.920467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.20.199.60:43703)
0925 14:57:56.920505 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.920507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.922986 (+ 2479us) server_negotiation.cc:407] Connection header received
0925 14:57:56.923077 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.923078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.923089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.923120 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.924730 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.925386 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.928194 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.928487 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.928533 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.930602 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.930603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.930604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.930611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.933893 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.934495 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.934495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.934496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.934614 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.934856 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.934857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.934857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.934976 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.935062 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.935283 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.935371 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:57:56.936599 11262 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.936671 11262 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:56.937004 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.937117 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:56.937434 11298 mvcc.cc:205] Tried to move back new op lower bound from 7333266542276857856 to 7333266542265724928. Current Snapshot: MvccSnapshot[applied={T|T < 7333266542276857856 or (T in {7333266542276857856})}]
I20260925 14:57:56.937482 11300 mvcc.cc:205] Tried to move back new op lower bound from 7333266542276857856 to 7333266542265724928. Current Snapshot: MvccSnapshot[applied={T|T < 7333266542276857856}]
I20260925 14:57:56.937448 11297 catalog_manager.cc:1347] Generated new cluster ID: c85bde9c6e3b4f7aa0ae6c4a1e31bfeb
I20260925 14:57:56.937554 11297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:56.949307 11181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } attempt: 3
Sep 25 14:57:56 dist-test-slave-8qt6 krb5kdc[11025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:56.960871 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.949642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:43703 (local address 127.0.0.1:57082)
0925 14:57:56.949745 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.949756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:56.949798 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:56.950045 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:56.950046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:56.950255 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:56.950322 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.950325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.951276 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.951278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.952093 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:56.952094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.952560 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.952560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:56.952634 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.954460 (+ 1826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:56.954466 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:56.957303 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:56.959670 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.959671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.959671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.959900 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.960163 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:56.960164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:56.960164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:56.960236 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:56.960536 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:56.960537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:56.960598 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 14:57:56.960660 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:57:56.960761 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 14:57:56.961547 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.949874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.20.199.60:43703)
0925 14:57:56.949916 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.949918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.949924 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:57:56.949948 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.949949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.949958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.950002 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.950409 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.951175 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.952180 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.952476 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.952559 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.957523 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.957525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.957525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.957534 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.959556 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.959978 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.959979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.959980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.960079 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.960300 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.960301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.960301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.960434 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.960492 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.961257 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.961333 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:56.963581 11181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: Bootstrap starting.
I20260925 14:57:56.965369 11297 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:56.966432 11297 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:56.966774 11181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:56.967108 11181 log.cc:824] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:56.968422 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.924388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.20.199.62:45603)
0925 14:57:56.924428 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:56.924430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:56.934797 (+ 10367us) server_negotiation.cc:407] Connection header received
0925 14:57:56.934893 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:56.934894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:56.934906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:56.934936 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:56.938647 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.939517 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.949121 (+ 9604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:56.950159 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:56.950207 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:56.964098 (+ 13891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:56.964101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:56.964102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:56.964112 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:56.966157 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.967150 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.967152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.967152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.967213 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:56.967612 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:56.967613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:56.967613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:56.967741 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:56.967833 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:56.968218 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 14:57:56.968328 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:56.970172 11181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: No bootstrap required, opened a new log
I20260925 14:57:56.972424 11181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.972569 11181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fa1bcda9084bbd9e595704d6910989, State: Initialized, Role: FOLLOWER
I20260925 14:57:56.972652 11181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } }
I20260925 14:57:56.972986 11181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:56.973810 11154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:56.974411 11297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649: Generated new TSK 0
I20260925 14:57:56.974586 11297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:56.974830 11154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [term 1 FOLLOWER]: Refusing update from remote peer 1ea3251d06424bd09078045a36308649: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:56.975078 11155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ea3251d06424bd09078045a36308649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "11fa1bcda9084bbd9e595704d6910989"
I20260925 14:57:56.975349 11276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:56.975816 11155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea3251d06424bd09078045a36308649 in term 1.
I20260925 14:57:56.977596 11318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.977705 11318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:56.977664 11305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea3251d06424bd09078045a36308649. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.977870 11305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:56.977759 11318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:56.981678 11318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:56.983920 11312 mvcc.cc:205] Tried to move back new op lower bound from 7333266542481674240 to 7333266542265724928. Current Snapshot: MvccSnapshot[applied={T|T < 7333266542276857856}]
I20260925 14:57:56.990151 11320 catalog_manager.cc:1259] Loaded cluster ID: c85bde9c6e3b4f7aa0ae6c4a1e31bfeb
I20260925 14:57:56.990208 11320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: loading cluster ID for follower catalog manager: success
I20260925 14:57:56.991345 11320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: acquiring CA information for follower catalog manager: success
I20260925 14:57:56.991703 11320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11fa1bcda9084bbd9e595704d6910989: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:57:57.009313 11185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.007308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.20.199.61:44147)
0925 14:57:57.007383 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.007385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.008624 (+ 1239us) server_negotiation.cc:407] Connection header received
0925 14:57:57.008733 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.008735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.008745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.008776 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.009241 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56052: BlockingRecv error: recv got EOF from 127.0.0.1:56052 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 14:57:57.010285 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.007934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.20.199.60:43703)
0925 14:57:57.008000 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.008002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.009537 (+ 1535us) server_negotiation.cc:407] Connection header received
0925 14:57:57.009719 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.009721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.009731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.009761 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.010227 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57088: BlockingRecv error: recv got EOF from 127.0.0.1:57088 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 14:57:57.010934 11109 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.008350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.20.199.62:45603)
0925 14:57:57.008391 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.008393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.009846 (+ 1453us) server_negotiation.cc:407] Connection header received
0925 14:57:57.010484 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.010485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.010495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.010524 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.010872 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35078: BlockingRecv error: recv got EOF from 127.0.0.1:35078 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:57.015928 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.012538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.20.199.61:44147)
0925 14:57:57.012571 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.012573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.012820 (+ 247us) server_negotiation.cc:407] Connection header received
0925 14:57:57.012935 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.012936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.012974 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.013016 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.013206 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.013996 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.015048 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.015299 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.015347 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.015559 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.015679 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.015726 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.015833 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.015849 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:57:57.019037 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.018242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.20.199.60:43703)
0925 14:57:57.018277 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.018278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.018512 (+ 234us) server_negotiation.cc:407] Connection header received
0925 14:57:57.018616 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.018617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.018634 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.018663 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.018983 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57102: BlockingRecv error: recv got EOF from 127.0.0.1:57102 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:57.024219 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.021099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.20.199.61:44147)
0925 14:57:57.021130 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.021132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.021371 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:57:57.021481 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.021482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.021490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.021516 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.021668 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.022389 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.023464 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.023748 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.023786 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.023980 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.024011 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.024036 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.024133 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.024152 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:57:57.027189 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.026458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.20.199.60:43703)
0925 14:57:57.026492 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.026494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.026710 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:57:57.026801 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.026802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.026812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.026840 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.027143 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57108: BlockingRecv error: recv got EOF from 127.0.0.1:57108 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:57.033270 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.030121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.20.199.61:44147)
0925 14:57:57.030157 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.030158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.030393 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:57:57.030483 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.030484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.030492 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.030518 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.030677 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.031440 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.032515 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.032778 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.032819 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.033029 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.033059 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.033086 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.033186 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.033202 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:57:57.036150 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.035402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.20.199.60:43703)
0925 14:57:57.035437 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.035439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.035654 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:57:57.035741 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.035741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.035751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.035779 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.036098 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57110: BlockingRecv error: recv got EOF from 127.0.0.1:57110 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:57:57.044392 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.041194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.20.199.61:44147)
0925 14:57:57.041229 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.041231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.041496 (+ 265us) server_negotiation.cc:407] Connection header received
0925 14:57:57.041589 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.041590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.041599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.041626 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.041783 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.042573 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.043635 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.043897 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.043941 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.044147 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.044178 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.044204 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.044304 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.044321 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:57:57.047526 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.046694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.20.199.60:43703)
0925 14:57:57.046739 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.046741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.046994 (+ 253us) server_negotiation.cc:407] Connection header received
0925 14:57:57.047095 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.047096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.047107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.047137 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.047477 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57116: BlockingRecv error: recv got EOF from 127.0.0.1:57116 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:57:57.060135 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.056699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.20.199.61:44147)
0925 14:57:57.056742 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.056744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.057054 (+ 310us) server_negotiation.cc:407] Connection header received
0925 14:57:57.057159 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.057160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.057170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.057200 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.057446 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.058189 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.059306 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.059599 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.059645 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.059851 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.059896 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.059927 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.060038 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.060058 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:57:57.063366 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.062531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.20.199.60:43703)
0925 14:57:57.062571 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.062573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.062801 (+ 228us) server_negotiation.cc:407] Connection header received
0925 14:57:57.062914 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.062915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.062927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.062956 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.063310 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57122: BlockingRecv error: recv got EOF from 127.0.0.1:57122 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:57.084123 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.080637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.20.199.61:44147)
0925 14:57:57.080686 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.080688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.081016 (+ 328us) server_negotiation.cc:407] Connection header received
0925 14:57:57.081123 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.081124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.081136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.081167 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.081379 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.082168 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.083321 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.083595 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.083639 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.083839 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.083886 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.083914 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.084022 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.084041 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:57:57.087307 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.086362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.20.199.60:43703)
0925 14:57:57.086406 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.086408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.086711 (+ 303us) server_negotiation.cc:407] Connection header received
0925 14:57:57.086827 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.086828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.086840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.086880 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.087251 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57128: BlockingRecv error: recv got EOF from 127.0.0.1:57128 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:57.123123 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.120646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.20.199.61:44147)
0925 14:57:57.120697 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.120699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.120836 (+ 137us) server_negotiation.cc:407] Connection header received
0925 14:57:57.120910 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.120912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.120923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.121000 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.121142 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.121678 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.122440 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.122630 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.122707 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.122863 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.122891 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.122941 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.123009 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.123028 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:57:57.125726 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.124852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.20.199.60:43703)
0925 14:57:57.124904 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.124906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.125109 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:57:57.125182 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.125183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.125194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.125274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.125657 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57144: BlockingRecv error: recv got EOF from 127.0.0.1:57144 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:57:57.192955 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.190898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.20.199.61:44147)
0925 14:57:57.190964 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.190966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.190995 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:57:57.191064 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.191066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.191075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.191136 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.191251 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.191717 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.192412 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.192596 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.192653 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.192768 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.192794 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.192831 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.192881 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.192896 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 14:57:57.195354 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.194680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.20.199.60:43703)
0925 14:57:57.194750 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.194753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.194836 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:57:57.194905 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.194906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.194917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.194974 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.195276 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57150: BlockingRecv error: recv got EOF from 127.0.0.1:57150 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:57:57.326807 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.324567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.20.199.61:44147)
0925 14:57:57.324643 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.324645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.324681 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:57:57.324755 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.324755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.324764 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.324804 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.324929 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.325482 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.326211 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.326391 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.326442 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.326567 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.326593 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.326630 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.326686 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.326700 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 14:57:57.329078 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.328522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.20.199.60:43703)
0925 14:57:57.328577 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.328580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.328590 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:57.328645 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.328646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.328656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.328730 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.329026 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57160: BlockingRecv error: recv got EOF from 127.0.0.1:57160 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:57.393572 11276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ea3251d06424bd09078045a36308649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea3251d06424bd09078045a36308649" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 45603 } } peers { permanent_uuid: "11fa1bcda9084bbd9e595704d6910989" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 44147 } } peers { permanent_uuid: "bb4f0fbd700f448d8870c5e2a317825c" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 43703 } } }
I20260925 14:57:57.393721 11276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea3251d06424bd09078045a36308649 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:57.589200 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.586477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.20.199.61:44147)
0925 14:57:57.586594 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.586596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.586605 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:57.586631 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.586632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.586641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.586810 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:57.587101 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.587604 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.588432 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:57.588635 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:57.588697 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:57.588837 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:57.588864 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:57.588902 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:57.588969 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:57:57.588984 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 14:57:57.591988 11260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:57.591323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.20.199.60:43703)
0925 14:57:57.591411 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:57.591414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:57.591438 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:57:57.591470 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:57.591471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:57.591481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:57.591551 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:57.591922 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57162: BlockingRecv error: recv got EOF from 127.0.0.1:57162 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:57:57.905655 11273 catalog_manager.cc:1259] Loaded cluster ID: c85bde9c6e3b4f7aa0ae6c4a1e31bfeb
I20260925 14:57:57.905766 11273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: loading cluster ID for follower catalog manager: success
I20260925 14:57:57.906747 11273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: acquiring CA information for follower catalog manager: success
I20260925 14:57:57.907207 11273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb4f0fbd700f448d8870c5e2a317825c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:57:58.107975 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.105317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.20.199.61:44147)
0925 14:57:58.105505 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:58.105508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:58.105520 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:57:58.105550 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:58.105551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:58.105562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:58.105628 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:58.105798 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.106405 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.107271 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.107459 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.107508 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:58.107639 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:58.107666 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:58.107705 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:58.107801 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:57:58.107820 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260925 14:57:58.112124 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.109788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.20.199.60:43703)
0925 14:57:58.109903 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:58.109905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:58.109935 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:57:58.109990 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:58.109990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:58.110055 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:58.110122 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:58.110227 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.110758 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.111430 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.111633 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.111688 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:58.111825 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:58.111937 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:58.111993 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:58.112044 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:57:58.112063 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260925 14:57:58.116699 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.113744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.20.199.62:45603)
0925 14:57:58.113846 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:58.113848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:58.113981 (+ 133us) server_negotiation.cc:407] Connection header received
0925 14:57:58.114084 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:58.114085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:58.114136 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:58.114217 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:58.114306 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.114821 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.115774 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.115977 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.116068 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:58.116268 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:58.116393 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:58.116464 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:58.116563 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:57:58.116587 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
I20260925 14:57:58.116947 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.113957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.20.199.60:43703)
0925 14:57:58.113994 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:58.113996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:58.114093 (+ 97us) server_negotiation.cc:407] Connection header received
0925 14:57:58.114167 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:58.114168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:58.114177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:58.114224 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:58.114386 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.115082 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.116150 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.116408 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.116448 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:58.116692 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:58.116722 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:58.116751 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:58.116865 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:57:58.116890 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:57:58.117051 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.113728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.20.199.61:44147)
0925 14:57:58.113776 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:58.113778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:58.113915 (+ 137us) server_negotiation.cc:407] Connection header received
0925 14:57:58.114023 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:58.114024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:58.114036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:58.114080 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:58.114255 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.114979 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.116166 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:58.116493 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:58.116553 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:58.116769 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:58.116804 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:58.116852 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:58.116942 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:57:58.116967 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:58.118856 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 11038
I20260925 14:57:58.124835 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 11113
I20260925 14:57:58.130151 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 11188
2026-09-25T14:57:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2839 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:58 dist-test-slave-8qt6 krb5kdc[11457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:58 dist-test-slave-8qt6 krb5kdc[11457](info): set up 2 sockets
Sep 25 14:57:58 dist-test-slave-8qt6 krb5kdc[11457](info): commencing operation
krb5kdc: starting...
W20260925 14:57:59.165689 21276 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/test-user.keytab.
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.199.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
Entry for principal kudu/127.20.199.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab.
I20260925 14:57:59.284938 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:36955
--webserver_interface=127.20.199.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:36955,127.20.199.61:42993,127.20.199.60:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:59.365063 11470 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:59.365278 11470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:59.365294 11470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:59.365314 11470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:59.366472 11470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:59.366515 11470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:59.366529 11470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:59.366542 11470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:59.366554 11470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:59.367936 11470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:36955,127.20.199.61:42993,127.20.199.60:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.62.keytab
--principal=kudu/127.20.199.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.62:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.199.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:59.368117 11470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:59.368284 11470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:59.373116 11475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:59.373134 11478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:59.373134 11476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:59.373188 11470 server_base.cc:1034] running on GCE node
I20260925 14:57:59.373566 11470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:59.373883 11470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:59.375053 11470 hybrid_clock.cc:648] HybridClock initialized: now 1790348279375029 us; error 38 us; skew 500 ppm
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:59.377472 11470 init.cc:377] Logged in from keytab as kudu/127.20.199.62@KRBTEST.COM (short username kudu)
I20260925 14:57:59.377878 11470 webserver.cc:460] Webserver started at http://127.20.199.62:36395/ using document root <none> and password file <none>
I20260925 14:57:59.378120 11470 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:59.378191 11470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:59.378331 11470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:59.379438 11470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data/instance:
uuid: "f291ad484cde4d37a842ecab0dfcd6ee"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "19175e4678a01bdd53088c1966339c31"
server_key_iv: "0f0dc6b89a03c7370568cb708c6d6772"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.379784 11470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal/instance:
uuid: "f291ad484cde4d37a842ecab0dfcd6ee"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "19175e4678a01bdd53088c1966339c31"
server_key_iv: "0f0dc6b89a03c7370568cb708c6d6772"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.381068 11470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:57:59.381793 11485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:59.381963 11470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:59.382071 11470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal
uuid: "f291ad484cde4d37a842ecab0dfcd6ee"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "19175e4678a01bdd53088c1966339c31"
server_key_iv: "0f0dc6b89a03c7370568cb708c6d6772"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.382143 11470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:59.390702 11470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:59.392416 11470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:59.392576 11470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:59.396359 11470 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.62:36955
I20260925 14:57:59.396404 11537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.62:36955 every 8 connection(s)
I20260925 14:57:59.396736 11470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:59.397778 11538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:59.399281 11538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.399430 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 11470
I20260925 14:57:59.399539 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-0/wal/instance
I20260925 14:57:59.399748 21276 external_mini_cluster.cc:1428] Setting key 333d746c528a31f77922a6334c19b61b
WARNING: no policy specified for kudu/127.20.199.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.61@KRBTEST.COM" created.
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:59.415772 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.400008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:36955 (local address 127.0.0.1:56220)
0925 14:57:59.400193 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.400235 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.400680 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.401102 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.401105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.401338 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.401549 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.401556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.402741 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.402743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.403911 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.403914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.404420 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.404422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.404513 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.410593 (+ 6080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.410603 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.412280 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.414213 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.414220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.414222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.414424 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.414690 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.414691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.414691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.414783 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.415157 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.415162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.415225 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.415337 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.415580 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":91,"thread_start_us":105,"threads_started":1}
I20260925 14:57:59.415772 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.400028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.20.199.62:36955)
0925 14:57:59.400821 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.400824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.400835 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:59.400937 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.400939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.400961 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.401058 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.401650 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.402640 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.404006 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.404312 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.404389 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.412380 (+ 7991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.412388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.412388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.412396 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.414099 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.414531 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.414534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.414535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.414618 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.414849 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.414850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.414851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.415081 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.415181 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.415377 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.415523 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":110,"threads_started":1}
I20260925 14:57:59.416491 11538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.416859 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.416734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39568)
0925 14:57:59.416785 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.416830 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:57:59.417027 11486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
I20260925 14:57:59.417575 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.417374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39578)
0925 14:57:59.417519 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.417554 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 14:57:59.417701 11538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:42993: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab.
Entry for principal kudu/127.20.199.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab.
I20260925 14:57:59.425525 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:42993
--webserver_interface=127.20.199.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab
--principal=kudu/127.20.199.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:36955,127.20.199.61:42993,127.20.199.60:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:59.466948 11538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } attempt: 1
I20260925 14:57:59.467422 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.467289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39592)
0925 14:57:59.467352 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.467391 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:57:59.467789 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.467690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39604)
0925 14:57:59.467737 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.467763 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:57:59.467882 11538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:42993: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
W20260925 14:57:59.505980 11545 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:59.506210 11545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:59.506228 11545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:59.506278 11545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:59.507442 11545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:59.507489 11545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:59.507505 11545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:59.507561 11545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:59.507588 11545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:59.509106 11545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:36955,127.20.199.61:42993,127.20.199.60:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.61.keytab
--principal=kudu/127.20.199.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.61:42993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.199.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:59.509370 11545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:59.509650 11545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:59.514570 11551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:59.514564 11553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:59.514595 11545 server_base.cc:1034] running on GCE node
W20260925 14:57:59.514564 11550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:59.515045 11545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:59.515300 11545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:59.516467 11545 hybrid_clock.cc:648] HybridClock initialized: now 1790348279516465 us; error 35 us; skew 500 ppm
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:59.519088 11545 init.cc:377] Logged in from keytab as kudu/127.20.199.61@KRBTEST.COM (short username kudu)
I20260925 14:57:59.519485 11545 webserver.cc:460] Webserver started at http://127.20.199.61:40123/ using document root <none> and password file <none>
I20260925 14:57:59.519723 11545 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:59.519791 11545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:59.519939 11545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:59.521092 11545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data/instance:
uuid: "8e916181269f4d43905cd1fbe270e130"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "a6a612162a1c358cd832a214f1b795ac"
server_key_iv: "4009ba0f012620eff7861da4960c6fc2"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.521550 11545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal/instance:
uuid: "8e916181269f4d43905cd1fbe270e130"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "a6a612162a1c358cd832a214f1b795ac"
server_key_iv: "4009ba0f012620eff7861da4960c6fc2"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.522939 11545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:59.523662 11560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:59.523825 11545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:59.523932 11545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal
uuid: "8e916181269f4d43905cd1fbe270e130"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "a6a612162a1c358cd832a214f1b795ac"
server_key_iv: "4009ba0f012620eff7861da4960c6fc2"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.524016 11545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:59.533617 11545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:59.535964 11545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:59.536036 11538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } attempt: 2
I20260925 14:57:59.536167 11545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:59.536527 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.536367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39608)
0925 14:57:59.536437 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.536482 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:57:59.536919 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.536799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39618)
0925 14:57:59.536852 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.536886 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:57:59.537040 11538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.61:42993: Network error: Client connection negotiation failed: client connection to 127.20.199.61:42993: connect: Connection refused (error 111)
I20260925 14:57:59.540392 11545 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.61:42993
I20260925 14:57:59.540442 11612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.61:42993 every 8 connection(s)
I20260925 14:57:59.540827 11545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/data/info.pb
I20260925 14:57:59.542066 11613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:59.543517 11613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:59.550028 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 11545
I20260925 14:57:59.550117 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-1/wal/instance
I20260925 14:57:59.550385 21276 external_mini_cluster.cc:1428] Setting key 8c8c383c00361fa6f218883edb9dbf86
I20260925 14:57:59.552340 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.544158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.20.199.62:36955)
0925 14:57:59.544212 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.544214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.544413 (+ 199us) server_negotiation.cc:407] Connection header received
0925 14:57:59.544855 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.544856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.544865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.544925 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.545546 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.546008 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.546939 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.547134 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.547186 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.549193 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.549194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.549195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.549201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.550956 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.551346 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.551347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.551347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.551410 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.551656 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.551657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.551657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.551815 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.551891 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.552211 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.552276 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:57:59.552640 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.544124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:36955 (local address 127.0.0.1:56224)
0925 14:57:59.544291 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.544333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.544770 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.544974 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.544977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.545219 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.545469 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.545473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.546088 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.546089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.546870 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.546872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.547212 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.547213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.547310 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.547743 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.547752 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.549091 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.551044 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.551052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.551053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.551270 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.551478 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.551479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.551479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.551589 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.551893 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.551900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.552070 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.552194 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.552367 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260925 14:57:59.553310 11613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.199.60@KRBTEST.COM; defaulting to no policy
I20260925 14:57:59.560962 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.553619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39620)
0925 14:57:59.553667 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.553678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.553717 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.553985 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.553986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.554152 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.554217 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.554219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.555067 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.555068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.555990 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.555993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.556461 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.556462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.556550 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.556917 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.556923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.558193 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.559928 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.559929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.559929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.560100 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.560315 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.560316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.560316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.560370 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.560653 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.560654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.560690 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.560754 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.560848 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:57:59.561030 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.553692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.20.199.61:42993)
0925 14:57:59.553841 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.553846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.553863 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:57:59.553885 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.553887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.553896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.553950 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.554300 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.554965 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.556082 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.556376 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.556435 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.558289 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.558297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.558297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.558304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.559804 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.560192 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.560195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.560195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.560249 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.560439 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.560441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.560441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.560583 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.560676 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.560782 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.560901 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.199.60@KRBTEST.COM" created.
I20260925 14:57:59.561705 11613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.562040 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.561923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59870)
0925 14:57:59.561973 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.562014 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:57:59.562198 11564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
I20260925 14:57:59.562587 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.562498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59884)
0925 14:57:59.562547 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.562571 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:57:59.562691 11613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:38675: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.199.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab.
Entry for principal kudu/127.20.199.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab.
I20260925 14:57:59.577286 21276 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyT1hpU/build/release/bin/kudu
/tmp/dist-test-taskyT1hpU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:38675
--webserver_interface=127.20.199.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab
--principal=kudu/127.20.199.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.199.20:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.199.62:36955,127.20.199.61:42993,127.20.199.60:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:59.611922 11613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } attempt: 1
I20260925 14:57:59.612447 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.612277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59892)
0925 14:57:59.612373 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.612403 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 14:57:59.612947 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.612693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59894)
0925 14:57:59.612855 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.612906 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
W20260925 14:57:59.613049 11613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:38675: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
I20260925 14:57:59.628244 11538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } attempt: 3
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:57:59.636134 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.628609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39622)
0925 14:57:59.628668 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.628680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.628735 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.628873 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.628874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.629043 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.629112 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.629114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.630046 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.630047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.630879 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.630882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.631289 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.631290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.631363 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.631857 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.631862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.633357 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.635087 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.635088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.635089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.635265 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.635485 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.635486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.635486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.635545 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.635819 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.635820 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.635870 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.635935 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.636032 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:57:59.636134 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.628647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.20.199.61:42993)
0925 14:57:59.628688 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.628690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.628717 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:57:59.628800 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.628801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.628811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.628868 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.629200 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.629951 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.630966 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.631211 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.631293 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.633542 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.633543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.633544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.633551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.634972 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.635355 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.635356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.635357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.635416 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.635626 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.635627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.635627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.635741 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.635817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.635957 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.636032 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:59.636989 11538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.637367 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.637273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59898)
0925 14:57:59.637316 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.637346 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:57:59.637734 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.637639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59912)
0925 14:57:59.637689 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.637713 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:57:59.637874 11538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:38675: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
I20260925 14:57:59.669076 11538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } attempt: 1
I20260925 14:57:59.669692 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.669516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59918)
0925 14:57:59.669602 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.669655 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:57:59.670092 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.669978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59928)
0925 14:57:59.670026 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.670071 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:57:59.670229 11538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:38675: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
W20260925 14:57:59.677306 11620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:57:59.677603 11620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:59.677642 11620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:59.677683 11620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:59.678783 11620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:59.678849 11620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:59.678874 11620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:59.678902 11620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:59.678932 11620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:59.680400 11620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.199.20:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.199.62:36955,127.20.199.61:42993,127.20.199.60:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/krb5kdc/kudu_127.20.199.60.keytab
--principal=kudu/127.20.199.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.199.60:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.199.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:57:59.680675 11620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:59.680891 11620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:59.681188 11613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } attempt: 2
I20260925 14:57:59.681628 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.681494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59936)
0925 14:57:59.681565 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.681598 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:57:59.681945 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.681854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59944)
0925 14:57:59.681898 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.681923 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:57:59.682039 11613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.199.60:38675: Network error: Client connection negotiation failed: client connection to 127.20.199.60:38675: connect: Connection refused (error 111)
W20260925 14:57:59.685698 11627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:59.685705 11629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:59.685704 11626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:59.685905 11620 server_base.cc:1034] running on GCE node
I20260925 14:57:59.686137 11620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:59.686426 11620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:59.687593 11620 hybrid_clock.cc:648] HybridClock initialized: now 1790348279687574 us; error 28 us; skew 500 ppm
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:59.690021 11620 init.cc:377] Logged in from keytab as kudu/127.20.199.60@KRBTEST.COM (short username kudu)
I20260925 14:57:59.690496 11620 webserver.cc:460] Webserver started at http://127.20.199.60:45779/ using document root <none> and password file <none>
I20260925 14:57:59.690737 11620 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:59.690805 11620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:59.690953 11620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:59.691963 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data/instance:
uuid: "aa909adde821403b912ae5f989914a0d"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "06f892792973624fe0c32a9f1572e75d"
server_key_iv: "1dbbe4a3d0178ca90158b424be60dc25"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.692322 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal/instance:
uuid: "aa909adde821403b912ae5f989914a0d"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "06f892792973624fe0c32a9f1572e75d"
server_key_iv: "1dbbe4a3d0178ca90158b424be60dc25"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.693660 11620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:57:59.694366 11636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:59.694538 11620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:59.694633 11620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data,/tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal
uuid: "aa909adde821403b912ae5f989914a0d"
format_stamp: "Formatted at 2026-09-25 14:57:59 on dist-test-slave-8qt6"
server_key: "06f892792973624fe0c32a9f1572e75d"
server_key_iv: "1dbbe4a3d0178ca90158b424be60dc25"
server_key_version: "encryptionkey@0"
I20260925 14:57:59.694710 11620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:59.703127 11620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:59.704766 11620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:59.704918 11620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:59.708913 11620 rpc_server.cc:307] RPC server started. Bound to: 127.20.199.60:38675
I20260925 14:57:59.708945 11688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.199.60:38675 every 8 connection(s)
I20260925 14:57:59.709272 11620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/data/info.pb
I20260925 14:57:59.710409 11689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:59.712107 21276 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyT1hpU/build/release/bin/kudu as pid 11620
I20260925 14:57:59.712208 21276 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyT1hpU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348014035135-21276-0/minicluster-data/master-2/wal/instance
I20260925 14:57:59.712038 11689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.712508 21276 external_mini_cluster.cc:1428] Setting key 2cd2b85303594865cae900b53f58cd77
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:59.721454 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.712886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.20.199.62:36955)
0925 14:57:59.712943 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.712945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.713148 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:57:59.713736 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.713737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.713749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.713794 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.714360 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.714836 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.715712 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.715902 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.715989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.718309 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.718310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.718310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.718318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.719946 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.720475 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.720476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.720476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.720528 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.720780 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.720781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.720781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.720877 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.720925 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.721234 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.721316 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:57:59.721611 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.712846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.62:36955 (local address 127.0.0.1:56230)
0925 14:57:59.713017 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.713057 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.713622 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.713822 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.713825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.714113 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.714260 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.714264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.714937 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.714938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.715610 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.715612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.716016 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.716017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.716126 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.716646 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.716656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.718215 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.720029 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.720036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.720038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.720396 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.720589 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.720590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.720591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.720712 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.720946 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.720952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.721097 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.721178 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.721301 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260925 14:57:59.722209 11689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.722287 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.712951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.20.199.60:38675)
0925 14:57:59.713127 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.713131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.713166 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:57:59.713633 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.713646 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.713668 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.713775 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.714414 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.715184 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.715930 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.716222 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.716337 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.718972 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.718980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.718980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.718986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.720953 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.721329 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.721331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.721331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.721385 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.721648 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.721650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.721650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.721806 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.721950 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.722061 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.722133 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.62@KRBTEST.COM
I20260925 14:57:59.728917 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.722481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.61:42993 (local address 127.0.0.1:39634)
0925 14:57:59.722590 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.722599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.722642 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.722763 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.722764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.722892 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.722918 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.722919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.723681 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.723682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.724171 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.724172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.724521 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.724522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.724581 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.724921 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.724926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.726254 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.727893 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.727894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.727894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.728098 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.728314 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.728315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.728315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.728376 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.728653 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.728654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.728704 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.728776 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.728861 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 14:57:59.728992 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.722551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.20.199.61:42993)
0925 14:57:59.722598 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.722601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.722631 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:57:59.722691 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.722692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.722704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.722760 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.723048 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.723609 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.724246 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.724443 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.724495 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.726339 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.726340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.726341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.726350 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.727815 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.728187 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.728189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.728189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.728245 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.728442 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.728443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.728443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.728584 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.728649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.728812 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.728882 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:59.729552 11689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:59.730584 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.723039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.20.199.62:36955)
0925 14:57:59.723076 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.723078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.723085 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:59.723148 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.723149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.723157 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.723208 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.723613 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.724075 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.724973 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.725146 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.725201 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.727819 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.727821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.727821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.727828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.729471 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.729877 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.729879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.729879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.729941 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.730148 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.730149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.730149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.730277 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.730437 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.730467 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.730526 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.60@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.199.61@KRBTEST.COM
I20260925 14:57:59.736586 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.729848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59958)
0925 14:57:59.729889 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.729898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.729934 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.730080 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.730081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.730237 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.730268 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.730269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.731216 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.731216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.731671 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.731672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.732059 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.732060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.732107 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.732506 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.732511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.733783 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.735401 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.735402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.735402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.735587 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.735906 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.735907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.735907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.735964 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.736261 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.736262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.736315 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.736399 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.736518 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 14:57:59.736711 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.729868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.20.199.60:38675)
0925 14:57:59.729921 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.729923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.729930 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:59.729995 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.729996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.730007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.730069 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.730342 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.731127 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.731763 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.731975 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.732059 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.733874 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.733875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.733875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.733881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.735291 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.735685 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.735686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.735686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.735781 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.736036 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.736037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.736037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.736175 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.736228 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.736433 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.736588 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":36}
I20260925 14:57:59.738490 11689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: Bootstrap starting.
I20260925 14:57:59.738662 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.731260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.20.199.61:42993)
0925 14:57:59.731294 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.731296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.731335 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:57:59.731404 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.731405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.731414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.731467 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.731919 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.732358 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.733067 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.733248 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.733301 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.735359 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.735360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.735361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.735368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.737549 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.737892 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.737893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.737894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.737958 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.738165 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.738166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.738167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.738300 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.738466 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.738504 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.738578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:57:59.739584 11689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:59.739900 11689 log.cc:824] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:59.740449 21276 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:59.740779 11689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: No bootstrap required, opened a new log
I20260925 14:57:59.741581 11512 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f291ad484cde4d37a842ecab0dfcd6ee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56238
I20260925 14:57:59.742533 11689 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.742681 11689 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa909adde821403b912ae5f989914a0d, State: Initialized, Role: FOLLOWER
I20260925 14:57:59.742769 11689 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.743052 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.743109 11689 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:59.743162 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:57:59.744928 11706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:59.744982 11706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:59.745379 11538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } attempt: 2
I20260925 14:57:59.745222 11620 master_runner.cc:186] Error getting master registration for 127.20.199.62:36955: OK, instance_id { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" instance_seqno: 1790348279395249 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:59.747573 11620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.62@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:59.753425 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.745690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.20.199.60:38675)
0925 14:57:59.745750 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.745752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.745798 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:57:59.745859 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.745860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.745872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.745954 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.746262 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.747291 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.748287 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.748603 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.748673 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.750793 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.750794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.750794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.750800 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.752249 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.752605 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.752606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.752606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.752664 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.752867 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.752868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.752868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.752996 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.753078 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.753232 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.753308 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:57:59.753400 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.745638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59968)
0925 14:57:59.745734 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.745745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.745789 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.745956 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.745957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.746124 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.746171 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.746174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.747398 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.747399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.748198 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.748202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.748688 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.748689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.748749 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.749128 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.749133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.750705 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.752341 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.752342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.752342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.752526 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.752731 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.752732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.752733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.752791 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.753069 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.753070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.753117 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.753208 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.753307 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 14:57:59.755447 11538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee: Bootstrap starting.
I20260925 14:57:59.756196 11538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:59.756508 11538 log.cc:824] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:59.757755 11538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee: No bootstrap required, opened a new log
I20260925 14:57:59.759159 11538 raft_consensus.cc:348] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.759308 11538 raft_consensus.cc:729] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f291ad484cde4d37a842ecab0dfcd6ee, State: Initialized, Role: FOLLOWER
I20260925 14:57:59.759402 11538 consensus_queue.cc:260] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.759708 11538 sys_catalog.cc:564] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:59.759668 11709 sys_catalog.cc:455] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.759742 11512 raft_consensus.cc:482] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:57:59.759759 11709 sys_catalog.cc:458] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:59.759799 11512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:59.760532 11512 raft_consensus.cc:504] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.760761 11512 leader_election.cc:288] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [CANDIDATE]: Term 1 election: Requested vote from peers 8e916181269f4d43905cd1fbe270e130 (127.20.199.61:42993), aa909adde821403b912ae5f989914a0d (127.20.199.60:38675)
I20260925 14:57:59.761044 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa909adde821403b912ae5f989914a0d"
I20260925 14:57:59.761168 11663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:59.762188 11663 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f291ad484cde4d37a842ecab0dfcd6ee in term 1.
I20260925 14:57:59.762759 11489 leader_election.cc:302] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa909adde821403b912ae5f989914a0d, f291ad484cde4d37a842ecab0dfcd6ee; no voters:
I20260925 14:57:59.762897 11709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:59.763044 11709 raft_consensus.cc:686] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [term 1 LEADER]: Becoming Leader. State: Replica: f291ad484cde4d37a842ecab0dfcd6ee, State: Running, Role: LEADER
I20260925 14:57:59.763149 11709 consensus_queue.cc:237] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.763553 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f291ad484cde4d37a842ecab0dfcd6ee. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.763778 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:59.762113 11470 master_runner.cc:186] Error getting master registration for 127.20.199.61:42993: OK, instance_id { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" instance_seqno: 1790348279539377 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:59.764197 11470 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:59.764127 11728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:59.764789 11728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:59.766980 11663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [term 1 FOLLOWER]: Refusing update from remote peer f291ad484cde4d37a842ecab0dfcd6ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:59.767333 11709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:59.768643 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f291ad484cde4d37a842ecab0dfcd6ee. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.768738 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:59.773154 11613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } attempt: 3
I20260925 14:57:59.774591 11709 sys_catalog.cc:455] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.774678 11709 sys_catalog.cc:458] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:59.775175 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.775275 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:59.777163 11733 mvcc.cc:205] Tried to move back new op lower bound from 7333266553920700416 to 7333266553910501376. Current Snapshot: MvccSnapshot[applied={T|T < 7333266553920700416}]
Sep 25 14:57:59 dist-test-slave-8qt6 krb5kdc[11457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.199.61@KRBTEST.COM for kudu/127.20.199.60@KRBTEST.COM
I20260925 14:57:59.779464 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.764285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.20.199.61:42993)
0925 14:57:59.764332 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.764334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.764343 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:59.764365 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.764366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.764386 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.764449 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.765726 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.766573 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.770521 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.770827 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.770873 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.773765 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.773767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.773767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.773774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.777060 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.778144 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.778145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.778146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.778222 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.778532 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.778533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.778533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.778672 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.778753 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.779056 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.779349 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:59.779930 11728 catalog_manager.cc:1347] Generated new cluster ID: 74cef633d8d44ec99731a49d0d186908
I20260925 14:57:59.779985 11728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:59.782833 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.773386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.199.60:38675 (local address 127.0.0.1:59976)
0925 14:57:59.773449 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.773459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:59.773500 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:59.773856 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:59.773857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:59.774046 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:59.774127 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.774128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.775725 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.775727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.776528 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:59.776530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.777211 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.777212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:59.777294 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.777730 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:59.777735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:59.779253 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:59.781800 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.781801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.781802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.781996 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.782197 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:59.782198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:59.782198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:59.782252 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:59.782545 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:59.782546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:59.782599 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:57:59.782645 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:57:59.782732 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:57:59.783417 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.773475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.20.199.60:38675)
0925 14:57:59.773664 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.773666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.773673 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:59.773694 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.773695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.773731 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.773822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.774198 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.775625 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.776629 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.777105 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.777552 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.779994 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.779995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.779996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.780002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.781707 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.782074 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.782075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.782075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.782129 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.782325 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.782326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.782326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.782457 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.782947 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.782992 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.783075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260925 14:57:59.784883 11613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: Bootstrap starting.
I20260925 14:57:59.785651 11613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:59.785995 11613 log.cc:824] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:59.788311 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.766748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.20.199.60:38675)
0925 14:57:59.766793 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.766795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.774837 (+ 8042us) server_negotiation.cc:407] Connection header received
0925 14:57:59.774941 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.774941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.774950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.774979 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.777259 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.777951 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.781511 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.781800 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.781839 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.784641 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.784642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.784642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.784660 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.786945 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.787435 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.787436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.787436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.787491 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.787723 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.787723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.787724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.787831 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.787890 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.788151 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.788231 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:59.790309 11613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: No bootstrap required, opened a new log
I20260925 14:57:59.790508 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.765647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.20.199.62:36955)
0925 14:57:59.765688 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.765690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.773652 (+ 7962us) server_negotiation.cc:407] Connection header received
0925 14:57:59.773756 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.773757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.773767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.773807 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.775568 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.776278 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.781930 (+ 5652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:59.782241 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:59.782305 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:59.784808 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:59.784809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:59.784809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:59.784816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:59.786531 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.786983 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.786984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.786984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.787051 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:59.789659 (+ 2608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:59.789660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:59.789660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:59.789810 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:59.789902 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:59.790335 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 14:57:59.790418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:59.792666 11613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.793330 11613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e916181269f4d43905cd1fbe270e130, State: Initialized, Role: FOLLOWER
I20260925 14:57:59.793594 11613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } }
I20260925 14:57:59.794230 11587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e916181269f4d43905cd1fbe270e130"
I20260925 14:57:59.794168 11613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:59.794519 11586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:59.795454 11586 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [term 1 FOLLOWER]: Refusing update from remote peer f291ad484cde4d37a842ecab0dfcd6ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:59.795863 11709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:59.794067 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.796314 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:59.796345 11587 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f291ad484cde4d37a842ecab0dfcd6ee in term 1.
I20260925 14:57:59.797344 11728 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:59.798175 11728 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:59.798197 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f291ad484cde4d37a842ecab0dfcd6ee. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.798261 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:59.798297 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.798332 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:59.799521 11755 catalog_manager.cc:1259] Loaded cluster ID: 74cef633d8d44ec99731a49d0d186908
I20260925 14:57:59.799563 11755 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: loading cluster ID for follower catalog manager: success
I20260925 14:57:59.800680 11755 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: acquiring CA information for follower catalog manager: success
W20260925 14:57:59.800813 11755 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 14:57:59.819445 11709 sys_catalog.cc:455] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f291ad484cde4d37a842ecab0dfcd6ee" member_type: VOTER last_known_addr { host: "127.20.199.62" port: 36955 } } peers { permanent_uuid: "8e916181269f4d43905cd1fbe270e130" member_type: VOTER last_known_addr { host: "127.20.199.61" port: 42993 } } peers { permanent_uuid: "aa909adde821403b912ae5f989914a0d" member_type: VOTER last_known_addr { host: "127.20.199.60" port: 38675 } } }
I20260925 14:57:59.819571 11709 sys_catalog.cc:458] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:59.821827 11728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f291ad484cde4d37a842ecab0dfcd6ee: Generated new TSK 0
I20260925 14:57:59.822017 11728 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:57:59.849956 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.848578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.20.199.62:36955)
0925 14:57:59.848624 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.848626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.849228 (+ 602us) server_negotiation.cc:407] Connection header received
0925 14:57:59.849334 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.849335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.849346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.849377 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.849884 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56256: BlockingRecv error: recv got EOF from 127.0.0.1:56256 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 14:57:59.850750 11693 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.848935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.20.199.60:38675)
0925 14:57:59.848977 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.848979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.850151 (+ 1172us) server_negotiation.cc:407] Connection header received
0925 14:57:59.850316 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.850317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.850327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.850356 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.850691 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59984: BlockingRecv error: recv got EOF from 127.0.0.1:59984 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:57:59.851682 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.851070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.20.199.61:42993)
0925 14:57:59.851115 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.851118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.851205 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:57:59.851296 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.851297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.851307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.851335 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.851624 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39672: BlockingRecv error: recv got EOF from 127.0.0.1:39672 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:57:59.858148 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.853667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.20.199.61:42993)
0925 14:57:59.853701 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.853703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.857593 (+ 3890us) server_negotiation.cc:407] Connection header received
0925 14:57:59.857708 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.857709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.857729 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.857758 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.858096 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39680: BlockingRecv error: recv got EOF from 127.0.0.1:39680 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:57:59.861116 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.860336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.20.199.61:42993)
0925 14:57:59.860369 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.860371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.860631 (+ 260us) server_negotiation.cc:407] Connection header received
0925 14:57:59.860722 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.860722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.860732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.860759 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.861071 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39684: BlockingRecv error: recv got EOF from 127.0.0.1:39684 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:57:59.864864 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.864179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.20.199.61:42993)
0925 14:57:59.864210 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.864212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.864404 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:57:59.864491 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.864492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.864501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.864529 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.864822 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39700: BlockingRecv error: recv got EOF from 127.0.0.1:39700 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:57:59.870759 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.870043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.20.199.61:42993)
0925 14:57:59.870077 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.870079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.870276 (+ 197us) server_negotiation.cc:407] Connection header received
0925 14:57:59.870371 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.870371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.870382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.870412 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.870713 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39710: BlockingRecv error: recv got EOF from 127.0.0.1:39710 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:57:59.880817 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.880008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.20.199.61:42993)
0925 14:57:59.880048 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.880049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.880281 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:57:59.880377 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.880378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.880390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.880417 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.880760 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39716: BlockingRecv error: recv got EOF from 127.0.0.1:39716 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:57:59.899206 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.898285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.20.199.61:42993)
0925 14:57:59.898330 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.898332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.898594 (+ 262us) server_negotiation.cc:407] Connection header received
0925 14:57:59.898704 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.898705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.898717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.898749 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.899149 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39720: BlockingRecv error: recv got EOF from 127.0.0.1:39720 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:57:59.941048 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:59.938821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.20.199.61:42993)
0925 14:57:59.938879 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:59.938881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:59.939418 (+ 537us) server_negotiation.cc:407] Connection header received
0925 14:57:59.939607 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:59.939608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:59.939667 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:59.939740 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:59.940182 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39722: BlockingRecv error: recv got EOF from 127.0.0.1:39722 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 14:58:00.007109 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:00.006417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.20.199.61:42993)
0925 14:58:00.006520 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.006522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.006533 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:00.006611 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.006612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.006623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.006680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:00.007023 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39728: BlockingRecv error: recv got EOF from 127.0.0.1:39728 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 14:58:00.137090 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:00.136383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.20.199.61:42993)
0925 14:58:00.136482 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.136484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.136496 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:58:00.136530 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.136531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.136542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.136615 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:00.137038 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39730: BlockingRecv error: recv got EOF from 127.0.0.1:39730 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 14:58:00.395164 11617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:00.394560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.20.199.61:42993)
0925 14:58:00.394617 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.394619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.394651 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:58:00.394726 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.394727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.394736 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.394791 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:00.395117 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39744: BlockingRecv error: recv got EOF from 127.0.0.1:39744 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:58:00.745455 11706 catalog_manager.cc:1259] Loaded cluster ID: 74cef633d8d44ec99731a49d0d186908
I20260925 14:58:00.745535 11706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: loading cluster ID for follower catalog manager: success
I20260925 14:58:00.746354 11706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: acquiring CA information for follower catalog manager: success
I20260925 14:58:00.746678 11706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa909adde821403b912ae5f989914a0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:58:00.801362 11755 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e916181269f4d43905cd1fbe270e130: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:58:00.911057 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:00.908359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.20.199.61:42993)
0925 14:58:00.908478 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.908481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.908546 (+ 65us) server_negotiation.cc:407] Connection header received
0925 14:58:00.908623 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.908624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.908685 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.908772 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:00.908921 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.909465 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.910242 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.910443 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.910529 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:00.910700 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:00.910808 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:00.910892 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:00.910935 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:58:00.910955 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260925 14:58:00.915032 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:00.912829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.20.199.60:38675)
0925 14:58:00.912949 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.912951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.912961 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:00.913006 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.913007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.913050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.913124 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:00.913203 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.913736 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.914416 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.914608 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.914656 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:00.914777 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:00.914867 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:00.914921 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:00.914955 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:58:00.914971 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260925 14:58:00.919436 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:00.916698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.20.199.61:42993)
0925 14:58:00.916746 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.916748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.916878 (+ 130us) server_negotiation.cc:407] Connection header received
0925 14:58:00.916981 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.916982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.916992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.917033 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:00.917179 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.917919 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.918762 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.918963 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.919038 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:00.919191 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:00.919221 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:00.919266 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:00.919321 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:58:00.919340 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:00.919989 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:00.916799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.20.199.60:38675)
0925 14:58:00.916837 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.916840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.917008 (+ 168us) server_negotiation.cc:407] Connection header received
0925 14:58:00.917031 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.917032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.917043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.917093 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:00.917258 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.917977 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.919105 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.919403 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.919476 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:00.919642 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:00.919672 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:00.919723 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:00.919874 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:58:00.919900 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:58:00.920219 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:00.916607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.20.199.62:36955)
0925 14:58:00.916802 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:00.916803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:00.916878 (+ 75us) server_negotiation.cc:407] Connection header received
0925 14:58:00.916965 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:00.916966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:00.917019 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:00.917099 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:00.917196 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.917950 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.919117 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:00.919427 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:00.919497 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:00.919761 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:00.919865 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:00.919923 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:00.920016 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:58:00.920037 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":55,"threads_started":1}
I20260925 14:58:00.921990 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 11470
I20260925 14:58:00.927079 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 11545
I20260925 14:58:00.931915 21276 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyT1hpU/build/release/bin/kudu with pid 11620
2026-09-25T14:58:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2802 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5641 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (266905 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST